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Chap ter_^ 


Intro duct  ion 


A  class  of  languages  called  LR  (k)  has  been  defined  by  Knuth 
[1965].  These  languages  may  be  deterministically  parsed  in  a 
left  to  right  fashion,  looking  at  most  "k”  symbols  ahead  of  the 
current  parse  point.  Knuth  described  a  method  for  generating  a 
recognizer  for  an  LR  (k)  grammar  that  consisted  of  a  deterministic 
pushdown  automaton.  Subsequent  work  by  Korenjak  [1969]  and 
DeRemer  [1969]  improved  upon  Knuth*s  work,  and  provided  efficient 
algorithms  for  generating  practical  parsers  for  programming 
languages . 

In  response  to  an  open  problem  by  Knuth,  Pager  [1970]  has 
proposed  the  technique  of  reducing  the  deterministic  finite  state 
controller  of  an  LR{k)  parser  by  applying  incompletely  specified 
finite  state  machine  (FSM)  reduction  theory  [Pauli  and  Unger 
1959].  In  doing  so,  the  large  number  of  states  produced  by 
Knuth*s  generator  for  non-trivial  LR  (k)  grammars  could  be 
substantially  reduced  in  number,  thus  making  Knuth* s  LR{k) 
algorithm  more  attractive  as  a  practical  technique.  DeRemer’s 
work  largely  obviates  Pager's  result  because  reasonably-sized 
parsers  may  be  more  readily  produced  by  his  technique  than  by  a 
combination  of  Knuth's  and  Pager*s. 

Pager  has  pointed  out  that  tie  minimization  of  the  number  of 
states  in  a  PSM  parser  controller  can  lead  to  a  substantial 
saving  of  space  in  the  coded  representation  of  the  parser  tables. 
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However,  this  saving  is  not  without  qualification:  the  detection 
of  errors  in  the  text  to  be  parsed  becomes  less  precise  with  the 
reduced  parser.  To  avoid  accepting  an  invalid  sentence,  the  top 
of  the  parse  stack  must  be  matched  with  the  symbols  of  the  right- 
hand-side  of  the  production  to  be  applied  before  a  reduction  is 
performed.  This  check  is  required  because  invalid  symbols  may 


have  been  stacked 

duri ng 

a  parse  sequence. 

Whereas 

the 

correctness  of  the 

parse 

stack  information 

is  assured 

in  an 

unminimized  LE  (k)  parser,  this  is  not  the  case  in  a  parser 
minimized  by  traditional  techniques. 


Aho  and  Oilman  [1972a]  have  specified  conditions  that  allow  a 
canonical  LR(k)  parser  to  be  minimized,  and  still  maintain  the 
error  detection  capability  of  the  unreduced  parser.  Their 
techniques  are  thus  more  generally  applicable  than  Pager's. 
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tation  of  an  IE  (k)  parser  as  a  reduced  matrix 
lementation  purposes  contrasts  with  the  more 
cture  representation  employed  in  implementations 
enerators  [Lalonde  1971,  DeRemer  1969,  Anderson 
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parser  generator  i nplemen ta tions  have  usually  opted  in  fa 
the  smaller  slower  list  structure  parsers  rather  than  th 
faster  matrix  structure  parsers.  Their  motives  for  this 
probably  were  based  on  the  lack  of  available  ccmpu 


orage  for  parser 

tables  in  operational 

compile 

rs 

ohibitive  storage 

requirement  of  large 

grammar 

pars 

trices,  and  not  on 

the  desire  to  minimize 

overall 
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decision 
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and  the 
er  table 
pilation 


In  this  thesis,  we  investigate  a  reduced  matrix  LR  (k 
structure  that  preserves  the  same  error  detection  capafcil 
an  unreduced  IB(k)  parser.  In  Chapter  2,  the  formalism  n 
for  the  development  of  a  suitable  error-preserving  reduce 
parser  is  presented.  These  techniques  are  more  gene 
those  of  Aho  and  Oilman,  and  more  widely  applicable.  In 
3,  algorithms  are  presented  that  allow  reduced  parser  mat 
be  generated  in  a  non-enumera five  manner.  In  Chapter 
practical  properties  of  the  matrix  reduction  algori 
examined,  and  the  table  space  requirements  and  parsing  e 


timings 

are 

develope  d 

and  compared 

with  existing  list  s 

parsers. 

We 

show  that 

the  reduced 

matrix  parser  tab 

compe  tit 
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in  size 

with  existing  list  structur 

represen 

ta  ti 

ons,  and 

that  reduced 

matrix  parsers 

substantially  faster  than  list  structure  parsers  for 
grammar.  We  consider  in  Chapter  5  the  application  of 
matrix  parsers  to  lexical  analysis  to  demon str 
effectiveness  of  the  reduced  matrix  structure.  We  conclu 
the  representation  of  LR  (k)  parser  tables  in  a  reduce 
structure  is  viable,  practical,  and  economic. 
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Chap t er_2 


Theory 

We  define  an  extended  finite  autcmaton  (EFA)  to  be  a  4-tuple 
{S,I,S°#F}  where 

S  Is  a  finite  non-empty  set  of  states, 

1  is  a  finite  non-empty  set  of  symbols, 

S°  is  a  distinguished  state  called  the  start  state, 

F  =  {f  { j)  /  1<j<n},  where  f  ( j)  is  called  a 

property  set.  At  least  one  of  f(j)  must 
be  the  set  S,  and  the  rest  are  in  general 
different  non-empty  sets  of  properties  such  that 
S  x  I  ->  f  ( j)  . 

For  example,  if  we  define  f(1)  (which  will  be  denoted  as  f1 
for  convenience)  to  be  the  set  of  states  S,  f2  to  be  the  output 
symbol  property  set  {0,1},  and  I  to  be  the  input  symbol  set 
{0,1},  it  is  clear  that  the  EFA  definition  includes  the  class  of 
binary- input  binary-output  finite  state  machines  (FSM)  [ Hennie 
1968].  In  general,  if  f1  is  S,  and  f2  is  a  finite  non-empty  set 
0  of  output  symbols,  where  1 S }  —  n ,  1I|=m,  |0}=p,  and  ]F]-2,  then 
the  EFA  definition  includes  the  class  of  general  n-state  m-input 
p-output  FSM’s. 

Recent  work  [Knuth  1965]  has  shown  that  there  is  a  class  of 
languages  which  have  deterministic  push-down  automata  (D PDA) 
recognizers.  A  DPDA  is  essentially  an  EFA  controlling  a  stack 
(push-down  store),  which  can  be  articulated  by  some  f€F.  We  are 
interested  in  the  EFA  part  of  these  recognizers.  In  particular. 
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the  DP DA  recognizers  of  Knuth  and  DeRemer  [1969]  and  others  may 
be  characterized  by  the  particular  UFA 
[S,I,SO, ff ir f*,f 3}J 
where  S  is  the  parser  state  set, 

I  is  the  grammar  vocabulary  set  of  terminal  and  non¬ 
terminal  symbols, 
so  is  the  start  state, 
f1  is  the  state  set  S, 

f2  is  the  output  symbol  set  composed  of 
{read,  look-ahead,  reduce} , 
f3  is  the  error  property  set  composed  of 
{no  error,  error} . 

TWe  have  explicitly  included  f3  as  an  errcr  property  since  in 
these  parsers  whenever  f 3  =  ”er tor’1 ,  f 1  and  f2  are  both  undefined, 
and  conversely  whenever  f3="no  error”,  f1  and  f2  are  both 
defined.  The  errcr  property  f3  may  be  factored  out  of  the  EFA 
into  an  error  function  ”E"  such  that  E  ( j , x)  =  M er ror"  if  and  only 
if  symbol  ”x"  is  known  not  to  occur  in  any  sentence  of  the 
language,  where  the  left  part  of  the  sentence  is  represented  by 
the  EFA  being  in  state  ” j",  and  E(j,x)=”nc  error”  otherwise. 
This  leaves  {f1,f2}  in  the  EFA.  It  is  apparent  that  the  factored 
EFA  representation  is  equivalent  to  that  for  the  FSIi  used  above. 
The  error  factored  EFA  may  be  shown  to  be  an  incompletely 
specified  (IS)  FSE ,  where  the  undefined  entry  at  (j,x)  represents 
a  "don't  care"  condition. 

For  convenience  we  extend  f1  to  S  x  I+  by 
f1  (j,xy)  =  f  1  (f  *  (j,x)  ,y) 
where  j  e  S,  x  €  I+,  and  y  £  I. 
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We  will  use  {fl,f2,f3}  as  they  are  defined  above  throughout 
this  presentation,  and  will  also  use  the  components  of  F  to 
represent  both  their  respective  mappings  of  Sxl  and  their 
respective  property  sets,  where  nc  ambiguity  should  result. 

ESM  Reduction  and  Equivalence 

Pauli  and  Unger  [  1959  ]  have  shown  that  the  number  of  states 
in  an  ISFSM  M  may  be  reduced  by  finding  a  closed  cover  Q  of  the 
state  set  S  such  that  1Q|<1S|,  where  components  g (i)  belonging  to 
Q,  (1<i<|Qj),  consist  of  maximal  state  set  classes  or  their 
subsets.  Their  technique  is  a  refinement  of  the  exhaustive 
enumeration  procedure  and  is  of  ccmbinat crial  complexity.  In 
order  to  apply  this  technique  in  practice,  it  is  desirable  that 
any  special  properties  of  the  EFA  be  exploited  since  we  will  be 

i 

dealing  with  EFA 1 s  of  several  hundred  states. 

If  Paull-Unger  reduction  were  applied  to  EFA  M  by  treating 
(f2,f  3)  as  an  output  pair,  theoretically  a  reduced  machine  H" 
could  be  produced.  However,  Pauli  and  Unger  have  also  shown  that 
by  assigning  values  to  unspecified  entries  of  an  ISFSK,  the  size 
of  the  reduced  M"  may  not  be  minimal.  Thus  consider  the  factored 
version  of  where  all  unspecified  entries  in  H*  are 
undefined.  We  seek  a  minimal  reduced  solution  M"  of  a 1  subject 
to  the  requirement  that  the  factored  composite  EFA  (Mn,E)  detect 
the  same  class  of  input  errors  at  the  same  parse  step  as  M  or 
eguivalently  as  (M*,E).  Thus  (M",E)  must  not  delay  detecting  an 
error  after  (H',b)  has  detected  an  error. 
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The  state  set  S  of  (M 1 , E)  is  defined  to  be  the  same  as  that 
of  (M”,E) .  The  EFA  (M”,E)  will  in  general  contain  a  mapping  of 
the  state  classes  of  M*  onto  those  of  M”. 

The  reduced  FSM  M"  must  be  capable  of  generating  a  suitable 
state  ”j"  at  every  parse  step  to  enable  the  deterministic 
generation  of  E(j,x)  for  every  symbol  nxM,  However,  the 
generation  of  a  reduced  FI”  does  not  necessarily  coincide  with 
deterministic  error  detection.  If  states  j  and  k  €  S  of  error- 
factored  ISFSM  M 1  of  EFA  n  are  merged  together  by  Paull-Unger 
reduction  into  one  state  "P”  of  reduced  FSM  M  ” ,  and  E  (  j,  x)  *E  ( k ,  x) 
for  some  x€I,  then  the  value  of  E{P,x)  is  undefined.  Thus  not 
every  reduced  closed  cover  M”  of  M *  forms  a  valid  solution.  The 
following  three  state  EFA  Ml  illustrates  this. 

1! 1 

A1  ,  ,1  C,0,0 

313,1,0  A, 0,0 
C | 3,0,0  C, 0, 0 
M  1 


Where  S=f  i=  {A,  B,  C} ,  f2={0,1],  f3={0,1},  I={x,y},  5  0  =  A .  In  the 

property  set  f3  the  value  0  corresponds  tc  Tlnc  error”,  and  1  to 
"error”.  Factored  into  Ml*  and  E,  this  becomes 
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A] 

C,0 

AJ 

1 

0 

B  IE,  1 

A ,  0 

B| 

0 

0 

C|B,0 

C,  0 

Cl 

0 

0 

Ml  ' 

E 

Using  Paull-Unger  reduction  on  Ml1  yields  a  two  state  realization 
Ml",  where  S=[U,V},  and  U={A,E)  and  V=(A,C}. 

When  the  symbol  nx"  is  read  by  MV’  in  V,  either  value  of  the 
error  property  may  correctly  be  produced  depending  on  the  state 
accessed  in  V.  If  state  A  was  accessed  in  V  then  E{V,x}-1; 
however,  if  state  C  was  accessed  then  E(V,x)=0.  It  is  apparent 
that  EFA  (M1",E)  is  not  equivalent  to  M=(M1*#E)  since  there  was 
no  error  ambiguity  in  Ml. 

The  following  section  presents  suitable  conditions  under 
which  the  EFA  (M",E)  is  guaranteed  to  be  equivalent  tc  fi. 

Minimization  Theory 

Following  the  work  of  Pauli  and  Unger  [1959]: 

Definition  1:  Two  states  j  and  k  of  an  error-factored  EFA  M*  are 
compatible  if  and  only  if  for  any  input  symbol  sequence, 
the  output  sequence  with  M*  initially  in  state  j  is 
identical  to  the  output  sequence  with  M 1  initially  in 
state  k  whenever  both  outputs  are  specified.  States  j 
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and  k  are  igco  mgajtgbjLe  if  they  are  not  compatible. 
{Eoth  j  and  k  are  members  of  S.) 

m 

We  adopt  the  special  symbol  U€M  for  the  compatibility  relation 
(overtyped  "C"  and 

From  this  definition  Grasselli  and  Luccio  [1955]  have  shewn 
that  states  j  and  k  are  compatible  if  and  only  if  for  every  input 
symbol  "x"  G  T 

f1(j,x)  €  f1(k,x)  whenever  both  are  specified, 
and  f2(j,x)  =  f2(k,x)  whenever  both  are  specified. 

Analagous  to  the  notion  of  an  Image  S'  of  a  set  of  compatible 

states  B,  where  S  *=  [f  1  ( j,  x)  ,¥  j£E}  for  seme  xGI,  consider  the 

following  new  definitions: 

Definition _ 2:  States  j  and  k  €  S  of  error-factored  EFA  M  1 

possess  a  potential  image  condition  if  and  only  if  for 
some  x€T  f 1  (  j , x)  1  (k,x)  ,  where  f  1  {j,x) ^  and  fl{k,x)#0. 

« 

Definition _ 3:  States  j  and  k  €  S  of  error-factored  EPA  M* 

possess  an  image  condition  if  and  only  if  for  some  x€T 

a)  3  €  k 

b)  j  and  k  possess  a  potential  image 

condition.  * 

Definition _ 4:  States  j  and  k  G  S  of  the  error-factored  E? A  H» 

are  error  equivalent  (EE)  if  and  only  if 

a)  j  €  k 

b)  E  ( j,  x)  =  E  (k,x)  ,  ¥  x  G  I, 

c)  f1  (j,x)  and  f1  (k,x)  are  also  EE  ¥  x  G  I, 
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where  f1(j»x)^p  and  fx(k,x)#p. 

Note  that  we  define  j  to  be  EE  with  itself.  * 

It  may  be  shown  that  this  definition  is  identical  to  that  of 
state  compatibility,  treating  {f2,f3}  as  the  output  function  in 
M 1  or  equivalently  in  M  .  Thus  consider  an  alternate  definition: 

Def initi on _ 5:  States  j  and  k  €  S  of  the  error-factored  EFA  M* 

are  successor  error  equivalent  (SEE)  if  and  only  if 

a)  j  €  k 

b)  E(f 1  (j,x)  ry)  =  E  (f 1  (k,x)  ,y)  ,  ¥  y  €  I,  and 

¥  x  €  I+  such  that  f1(j,x)7ip  and  fl{k,x)^p. 

If  j  and  k  are  not  SEE  then  they  are  successor 
error  distinguishable  (SED) .  « 

If  the  restriction  of  the  range  of  f3  (ie-  on  the  range  of 
error  function  values)  is  arbitrarily  changed  to  (no  error, 
error,  p} ,  where  p  represents  a  "don’t  care"  condition, 
definition  5  becomes: 

Definition _ 5._J:  States  j  and  k  e  S  of  the  error-factored  EFA  n » 

are  weak  SEE  if  and  only  if  they  are  SEE  by  definition 
5,  and  the  following  is  added  to  the  end  of  part  b)  of 
definition  5: 

when  E(fl{j,x),y)  *  p  and  E(f1(k,x),y)  #  p  « 

To  illustrate  the  difference  between  definitions  4  and  5, 
consider  the  following  EFA  M2  factored  into  (M2*,E). 


1  X  y 

1_ 

x  _ 

A|  B,0 

A! 

1 

0 

E|A,  1 

E| 

0 

1 

M2  •  E 
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In  M21,  states  A  and  B  are  SEE  but  not  EE,  If  M2  were 
considered  unfactcred,  states  A  and  B  of  M2  would  be  incompat ible 
since  the  output  pairs  {f2,E}  are  not  egual  when  specified.  It. 
is  evident  that  by  the  SEE  definition,  M2*  nay  be  reduced  to  one 
state  C  in  M2”: 

l x I 

C |A, 1  E  ,  0 

M2” 

Given  the  start  state  in  M2',  and  an  index  mapping  for  states  in 
M2*  to  states  in  M2”  (A->C,  E->C) ,  the  machine  (M2”,E)  could 
imitate  the  behaviour  of  M2  exactly. 

It  may  be  shown  that  the  SEE  relation  between  states  is  a 
compatibility  relation  [Prather  1967]  and  not  an  equivalence 
relation,  since  SEE  is  reflexive  and  symmetric,  but  not 
necessarily  transitive.  Likewise  the  EE  relation  is  a 
compatibility  relation.  (It  can  also  be  noted  that  EE  is  the 
c lassical  ISFSM  definition  of  state  equivalence.  Also,  two 
states  are  SEE  if  and  only  if  they  are  1 -cc mpatible  and  their 
successors  are  equivalent.)  Thus  following  Pauli  and  Onger  [1959] 
there  exists  a  minimum  reduced  state  version  M”  of  error  factored 
EFA  M '  composed  of  a  closed  cover  of  compatibility  classes. 
Their  result  is  not  dependent  upon  the  definition  of 
compatibility  employed  -  EE  or  SEE.  However,  if  (M”,E)  is  to 
detect  the  same  class  of  input  (syntactic)  errors  as  M,  it  must 
be  shown  that  SEE  and  EE  preserve  the  error  detection  capability 
of  M  in  ( M ” , E )  .  These  results  follow. 


I 


Theorem 


P  roof : 


Theorem 


Proof : 


_1:  If  M  =  (M',E)  is  an  EFA  of  an  LR(k)  grammar,  then  a 

reduced  ETA  (M",E)  may  be  formed  that  detects  the  same 
class  of  input  errors  as  M,  where  M"  is  composed  of  a 
closed  cover  of  EE  state  compatible  blocks  of  M*. 

The  proof  follows  directly  by  considering  the  unfactored 
EFA  M  with  a  two  function  output  {f2,E}.  By  Paull-Unger 
and  the  definition  of  EE,  there  does  not  exist  any 
distinguishing  sequence  of  length  "l"  such  that  the 
output  {f2,E}  from  the  reduced  closed  cover  is 
distinguishable  from  that  of  the  unreduced  machine  M. 

T\ 

2:  If  M  =  (M*,E)  Is  an  EFA  of  an  LB  (k)  grammar,  then  a 

reduced  EFA  (M” , E)  may  be  formed  that  detects  the  same 
class  of  input  errors  as  M,  where  M”  Is  composed  of  a 
closed  cover  of  SEE  state  compatible  blocks  of  H*,  and 

|  M" J<1 1 . 

We  need  only  show  that  every  transition  in  the  reduced 
machine  H*’  preserves  error  detectability.  The  following 
is  a  constructive  proof  for  the  generation  of  a  suitable 

Mn . 

Let  C  =  {C1,C2,  ...,Ci,  ...  ,Cr}  be  the  set  of  SEE  state 

compatible  blocks  of  M*,  where  Ci=  [s 1 , s2 , . . . , si}  and 
si, s2, ... sies .  Each  Ci  is  a  state  of  M". 

Partition  each  Ci  into  "e"  error  equivalence  classes 
Ci  =  {Ci 1 , Ci2 ,  ...  Cim} ,  where  m<]Cij, 

such  that  all  states  in  any  Cip  are  EE  (for  1<p<|Ci|). 
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Let  fl(Ci,x)  =  Cj  be  in  M”  for  seme  x£I  (for  1<j<r). 
Then  there  exists  a  unique  T'q"  such  that 
f 1 (Cip, x)  =  C jq 
where  1<g<|Cj],  1<p<|Ci). 

Define  a  mapping  ’'Index1'  such  that 
Index(Cij)  -  Ci,  ¥i,j. 

(ie-  Index  maps  each  SEE  compatibility  class  of  M f  that 
is  EE  into  its  state  in  H":  Index  ( f 1  (Cip,  x)  )  =  Cj.) 

Replace  all  transition  entries  fl(Ci,x)  ¥xGI  in  M”  by 
their  corresponding  Cjq  (1<q<|Cj|)  by  inspection. 

Each  EE  state  set  Cij  has  a  unique  error  function 
E(Cij,x)  ¥xGI,  where  1<j<jCij,  1<i<r. 

If  (M",E)  is  started  in  SEE  compatible  Cl,  where  S^>€C11 
is  the  start  state  of  M,  then  error  detection  is 
preserved  in  the  starting  state  if  the  error  equivalence 
class  for  Cl  1  is  E{S°,x)  ¥x€I. 

Thus  each  transition  of  the  reduced  M "  maps  into  an  SEE 
compatible  Ci,  and  produces  a  unique  error  equivalence 
class  number  j  (ie-  Cij)  that  may  be  used  for  error 
detection  in  the  compatible  fi(Cij,x)  ¥x8I. 

Both  the  SEE  and  the  EE  definitions  allow  for  the  production 
of  reduced  solution  EFA's  {K”,E)  that  detect  the  same  input 
errors  as  H.  The  question  of  which  definition  produces  the  more 
reduced  M”  is  partly  answered  by  the  following  lemmas. 

Lemma _ 1:  All  closed  covers  of  EE  compatible  blocks  are  SEE. 
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Proof :  The  proof  is  obvious  from  the  definition  of  EE  and  SEE. 

Lemma _ 2:  Not  all  closed  covers  cf  SEE  compatible  blocks  are  EE. 

Proof:  The  example  cf  EFA  K2  shows  that  the  converse  ’’All 

closed  covers  of  SEE  compatibles  are  EE”  is  not  in 

general  true,  thus  the  lemma  is  proved. 

Lemma _ 3:  The  set  of  closed  cover  machines  formed  from  EE 

compatibles  is  a  subset  of  the  set  of  closed  cover 
machines  cf  SEE  compatibles. 

P roof :  The  proof  is  obvious  from  Lemmas  1  and  2.  n 

From  these  lemmas  it  is  apparent  that  SEE  is  a  less 

restrictive  definition  than  EE.  Thus  the  class  of  reduced  SEE 
closed  covers  will  in  general  contain  more  solution  machines  than 
the  class  of  reduced  EE  closed  covers.  Consequently  we  adopt  SEE 
as  the  more  general  compatibility  definition  of  the  two. 

From  the  above,  if  SEE  states  are  merged  together  in 
compatible  blocks  of  EFA  (K”,E),  then  the  same  class  of  input 

errors  will  be  detected  as  EFA  M=(*1’,E),  where  jMn|<]M,|.  From 

Paull-Unger  H”  is  one  of  a  possible  multiplicity  of  state -min imal 
closed  covers  that  can  be  constructed  from  the  SEE  state 
compatible  blocks  of  E » .  Thus  (M",E)  is  the  smallest  reduced  EFA 
that  preserves  the  error  detecting  capability  of  tl={M,,E)  . 
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The  technique  of  Paull-Unger  may  be  applied  to  produce 

minimal  error- preserved  EFA’s  (M”,E)  from  ( M  1 ,  E ) .  However,  the 

SEE  criterion  does  not  reduce  the  inherent  computational  effort 
in  finding  a  solution  H”. 

Definition _ 6:  States  j  and  k  £  S  of  an  error-factored  ISFSH  M 1 

\ 

are  disjoint  compatible  (DC)  if  and  only  if  for  all 

symbols  x€I 

F  ( j,  x)  =  F  (k,x) 

whenever  F(j,x)  £  p  and  F{k,x)  *  0. 

If  j  and  k  are  not  DC  then  they  are  disjoint 
incompatible.  * 

We  represent  the  DC  relation  by  the  special  symbol  (overtyped 

T,CM  and  ,,=n)  .  This  definition  is  the  SEE  relation  for  the 

special  case  that  if  f1(j,x)#p  and  f1  (k  then  f  1  ( j  ,  x)  =f 1  { k ,  x) 

¥  x  €  I.  It  is  then  apparent  that  since  SEE  and  consequently  DC 
are  compatibility  relations  that  if  two  states  j  and  k  Q  S  of  an 
error-factored  ISESM  Mf  are  DC,  then  they  are  also  compatible. 

Using  DC  the  next  theorem  follows  directly: 

Theorem  3:  All  covers  composed  of  DC  state  blocks  of  an  ISESM  K 1 
are  closed. 

Proof:  let  C  be  any  cover  composed  of  DC  state  blocks  of  M * , 

where 

C  =  {Cl  ,C2,  ...  , Cr }  . 

From  the  DC  definition  it  follows  that  block  Ci  has  at 
most  a  singleton  specified  state  transition  successor 


-15- 


. 


for  any  input  xCI.  Because  C  covers  all  states  of  M 1 , 

the  successor  of  Cl,  f1  (Ci,x)  ,  is  contained  in  some 

< 

cover  block  Cj  of  C  provided  that  F  (Ci , •£)*$.  Thus  C 

forms  a  closed  cover  of  M 1 .  Since  C  is  any  set  of  DC 

state  cover  blocks  of  M*,  the  theorem  is  proved. 

n 

An  important  subset  of  the  set  of  all  covers  of  EC  blocks  is 
the  set  of  covers  composed  of  maximal  DC  blocks.  (A  block  is 
maximal  if  it  is  not  included  in  any  ether  larger  EC  block.)  It 
follows  that  all  covers  of  maximal  EC  state  blocks  of  M*  are 
closed . 

Since  the  DC  relation  is  a  special  case  of  the  SEE  relation, 
it  is  apparent  that  all  DC  state  block  covers  M"  of  M *  are  SEE. 

It  is  also  apparent  that  the  use  of  state  DC  instead  of  state 

compatibility  simplifies  the  Paull-Unger  reduction  technique  by 
eliminating  the  closure  requirement  of  their  covering-closure 
problem.  Deduced  closed  cover  FSM*s  E"  may  be  generated  for  H ’ 
that  detect  the  same  input  error  class.  However,  the  size  of  fl" 
C  3  E”  1 )  using  state  DC  may  exceed  that  using  state  compatibility. 
We  will  subsequently  show  that  the  use  of  DC  simplifies  the 
complexity  of  the  reduction  technique. 
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Ginsburg  [  1959]  has  shown  that  a  computational  estimate;  of 

the  lower  bound  of  ]Mn j  may  be  formed  by  determining  the  number 

< 

of  states  contained  in  the  largest  maximal  incompatible  block  of 
M ' .  If  DC  is  used  instead  of  compatibility,  a  lower  bound  on  the 
number  of  DC  state  blocks  is  the  number  of  states  in  the  largest 
maximal  disjoint  incompatible  block.  This  computational  lower 
bound  using  DC  will  be  referred  to  as  the  DC  lower  bound  (DCLB) , 
and  using  compatibility  as  the  Gins  burg  lower  bound  (GLB)  . 

Lemma  4:  For  EFA  E={iV,E),  GLB(M')  <  DCLE  (E  1 )  . 

Proof :  Since  the  DC  definition  is  a  subset  of  the  compatibility 

definition,  it  fellows  that  the  converse  of  the  above 
GLB  (M  1  )  >  DCLB  (M  * ) 

cannot  be  true.  Thus  the  lemma  Is  proven  by 
contradiction. 

SI 

Corollary:  GLB  { ft  1 )  <  WSFEIB  (M  * )  <  SEELE  (£!»)  <  DCLE  (M*  )  ,  where 

SEELB  (M 1 )  is  the  number  of  states  in  the  largest  maximal 
incompatible  block  of  SEE  states,  and  SSEELE  (M * )  Is  that 
for  weak  SEE  states. 

a 

Thus  reduced  cover  EFAfs  whose  compatible  state  blocks  are  formed 
under  the  SEE  or  the  DC  definitions  may  contain  more  blocks  than 
those  formed  under  the  compatibility  definition. 

The  DeRemer  characteristic  FSE  recognizer  [DeRemer  1969]  may 
be  represented  as  an  EFA  such  that  f1  is  the  state  set  S,  and 
f2={read,  look-ahead,  reduce}.  The  structure  of  the 
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characteristic  FSH  recognizer  M 1  of  EFA  M  is  such  that  each  state 
j  belonging  to  S  is  entered  after  reading  a  unique  (possibly 
null)  symbol:  ie-  all  transitions  to  state  j  appear  under  one  FSM 
symbol  column  in  the  transition  matrix.  Thus  the  sets  of  states 
appearing  under  any  FSM  symbol  x,  (fl(j/X)},  do  not  overlap  with 
those  of  {fMh,y)},  where  x*y ,  for  all  j*k  in  S.  Since  all 
states  in  M '  are  accessible  from  the  distinguished  state  S°  by 
the  LR  (k)  construction  algorithm  by  an  applicable  input  sequence 
of  symbols  from  set  I,  and  since  no  states  are  superfluous 
[DeRemer  1969],  only  a  limited  number  cf  different  states  can 
appear  in  [f1  (k,x),  ¥k}  .  Thus  f1(j,x),  for  j#k,  may  have 
potential  image  conditions  with  only  a  few  other  states  under 
symbol  x. 


We  now  strengthen  the  results  of  lemma  4  by  considering  the 
SLR(k)  or  the  LALR  (k)  FSM  generated  by  the  DeRemer  construction 
technique.  We  note  that  by  this  construction  the  FSM  produced 
for  an  SLR  (k)  or  an  LALR  (k)  grammar  has  the  following  properties 
(among  others) : 

a)  Each  characteristic  string  of  the  FSM  is  unique, 
and  has  a  unigue  reduce  transition  (ie-  a  unique 
production  number  output  function  value) ; 

b)  The  FSM  contains  no  superfluous  or  duplicated 
states. 

Consider  the  FSM  output  function  f2-(read,  look-ahead, 
reduce]  for  an  SLR  (k)  or  an  LALR (k)  grammar  G  of  up”  productions, 
such  that  if  production  "i"  is  to  be  applied  then  f 2 ( j , x) =red uce, 
and  f1(j/X)=i,  Property  a)  above  implies  that  all  states  j,  k  € 
S  of  the  FSM  state  set  having  reduce  transitions  (#j  and  #k 
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respectively)  are  distinguished  by  the  production  number,  unless 
#j=#k.  Ey  induction  on  the  length  of  applicable  sequences  of 
input  symbols  to  the  FSM  the  .following  lemma  is  true: 

Lemma _ 5:  If  states  j  and  k  €  S  cf  a  DePemer  SLR  or  LALP  FSM 

access  #j  and  #k  reduce  transitions  for  the  same 
applicable  sequence  of  input  symbols  X=  {x1 , x2, . .  .} ,  then 
j  and  k  are  distinguishable,  consequently  they  are 
incompa  ti ble . 

» 

Consider  states  j  and  k  C  S  of  the  error-factored  EFA  M 1 
formed  from  the  DePemer  SLR  or  LALR  FSM  of  grammar  G.  We  have 
shown  that  jDk  implies  that  j  and  k  are  SEE  (since  DC  is  a 
special  case  of  SEE) .  However,  we  intend  to  show  that  if  j  and  k 
are  SEE,  they  are  DC  for  this  class  of  machines.  This  implies 
that  the  state  class  that  is  "SEE  but  not  DC"  must  be  shown  to  be 
empty . 


We  note  first  that  the  above  relation  is  trivially  true  for 
reduce  transitions.  Thus  we  need  only  consider  non-empty  read 
and  look-ahead  transitions  from  compatible  states.  The 
definition  of  an  image  condition.  Definition  3,  describes  states 
in  this  class. 

Theorem 4:  If  j  and  k  £  S  of  the  error-factored  EFA  M 1  formed 

from  the  EeRemer  SLR/LALR  FSM  of  a  grammar  G  possess  an 
image  condition,  then  j  and  k  are  not  SEE. 
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L£22iL:  Ey  Definition  3,  j  £  k.  Thus,  by  Lemma  5  they  cannot 

access  #j  and  #k  respectively,  where  #j£#k.  (If  #j-$k 

< 

the  state  successors  of  j  and  k  under  the  image 
condition  f1  (j,x)  and  f*  (k,x)  respectively,  would  have 
been  merged  by  the  DeRemer  SLR/LALR  construction 
algorithm.)  Hence  there  must  be  some  non-empty 
successor  of  j  and  k,  say  y=fl{j,x)  and  z-f1  (k,x),  which 
by  Definition  3  implies  y=z  for  seine  xSX .  Eut  since  no 
states  are  superfluous,  y  and  z  must  access  different 
input  symbols:  ie-  E  (y , w) # E  (z , w)  for  some  wGI.  Eut  this 
contradicts  the  SEE  definition  for  states  j  and  k.  Thus 
j  and  k  are  not  SEE. 


Theorem  5:  States  j  and  k  Q  S  of  the  error-factored  EFA  M ’ 

formed  from  the  DeRemer  SLR  or  LAL'B  FSW  of  a  grammar  G 

are  SEE  if  and  only  if  they  are  DC. 

Proof :  For  the  "if”  part:  j£k  implies  j  and  k  are  SEE  by 

definition.  For  the  "only  If"  part:  j  and  k  are  SEE 
implies  that  jDk  by  Theorem  4. 

91 

The  above  results  mean  that  the  DC  relation  may  be  used 

instead  of  the  more  complex  SEE  relation  in  the  formation  of 
reduced  EFA’s  of  FSM’s  produced  by  DeRemer’s  method. 

Lemma  6:  For  an  EEA  M=(M,/E)  of  a  DeRemer  LALR  FSM 

SEELB  ( H 1 )  =  DCLE  (M  * )  . 

Corollary:  GLB  ( M  * )  <  W  SFELE  ( M  *  )  <  SEELB  (M*)  =  DCLE  { li  * ) 
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Thus  the  reduced  cover  EFAfs  whose  DC  state  blocks  form  the  cover 
contain  the  same  number  of  blocks  as  those  formed  under  the 
SEE  relation. 

£ revious  Work 

Pager  [  1970  ]  has  proposed  the  minimization  of  the  FSM  portion 
of  LR  (k)  recognizers.  However,  his  minimization  was  performed 
using  the  definition  cf  state  compatibility:  hence  his  reduced 
PSM’s  do  not  preserve  the  error  detecting  capability  of  the 
unreduced  FSM . 

Aho  and  Oilman  [1972a]  have  given  conditions  under  which 
their  version  of  the  Knuth  set  of  LR  (k)  parsing  tables  may  be 
reduced  while  preserving  its  ability  to  detect  errors.  Their 
first  algorithm  determines  which  cf  the  error  entries  in  the 
parsing  tables  may  be  replaced  by  ’’den1!  care”  entries  by 
determining  those  table  error  entries  that  can  never  be  accessed 
by  an  invalid  input  symbol  sequence.  This  "p-inaccessible”  set 
of  incompletely  specified  tables  is  subsequently  reduced  by 
compatible  partitioning  of  the  state  set,  yielding  a  reduced  set 
of  tables  equivalent  to  the  original  set.  We  compare  the  EFA 
(M?,E)  with  Aho  and  Ulliuan’s  fables  after  the  application  of 
their  first  algorithm. 

Lemma _ 7:  Let  M=(M»,E)  be  the  EFA  of  an  arbitrary  LR  (k)  grammar 

FSM,  and  let  T  be  the  corresponding  ^-inaccessible 
table.  Not  all  p  entries  in  M*  are  p  in  T. 
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:  We  show  one  instance  where  an  error  entry  in  T  may  bo 

replaced  by  a  J?  entry  in  M  * .  In  the  formation  of  the 

< 

initial  table  set  for  state  S°,  Aho  and  Ullman  stipulate 
that  the  entry  at  (SO,e)  must  not  be  replaced  by  p,  but 
must  remain  as  an  error  entry.  (The  symbol  Me”  is  the 
empty  string.)  However,  by  our  error-factorization 
technique,  the  error  indicator  for  this  entry  may  be 
factored  out.  and  placed  in  E(So,e).  Thus  in  our 
representation,  the  FSM  entry  at  (S°,e)  may  be  replaced 
by  0. 

We  require  that  the  validity  cf  a  particular  state  transition 
in  M*  be  verified  by  inspecting  the  error  indicator  prior  to 
accessing  the  transition/output  information.  Thus  when 
E  (j,x}=’'no  error11,  f  l{j,x)ttpt  and  f  2  { j  ,x)*p.  Hence,  we  may 
trivially  show  the  EEA  (M*,E)  is  (^-inaccessible  in  the  same  sense 
as  the  Aho  and  Ullman  tables.  The  results'  obtained  by  error¬ 
factoring  the  ETA  are  potentially  superior  to  those  obtained  by 
Aho  and  Ullman  in  that  M 1  contains  more  p  entries  than  T.  Thus 
the  reduced  TSH  K"  could  be  smaller  than  the  reduced  FSM  derived 
from  T. 

Aho  and  Oilman’s  first  algorithm  replaces  only  a  subset  of 
the  set  of  all  error  entries  in  T  by  0.  Ey  design,  all  the  error 
entries  in  H 1  have  been  factored  into  E,  and  replaced  by  jar . 
Since  prior  to  the  introduction  of  0  entries,  T=M,  all  Aho  and 
U lima n 1 s  p  entries  in  T  are  included  in  the  set  of  p  entries  in 
M  * .  The  0  entries  in  Aho  and  Oilman’s  p- inaccess ible  tables  thus 
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form  a  proper  subset  of  the  entries  in  the  err or- f act cred  ESM 

M  *  • 

The  definition  of  state  compatibility  used  by  Aho  and  Ullman 
in  the  formation  of  their  reduced  ^-inaccessible  tables  is  the 
weak  EE  relation  (ie-  EE  with  0  entries  in  the  error  function) . 
It  has  been  shown  that  the  set  of  reduced  EE  closed  covers  is 

contained  in  the  set  of  reduced  SEE  closed  covers.  Thus  the 
latter  set  may  possess  a  more  ninlmal  reduced  solution  H". 
Similarly  any  machine  that  has  been  reduced  using  weak  EE 
compatibility  may  also  possess  a  more  minimal  reduced  solution  M” 
using  weak  SEE  compatibility.  Consequently 
GIB  (T)  =  WEELE  (M * )  >  RSEELB  (M * ) 

where  GLE(T)  is  the  Ginsburg  size  lower  bound  of  the  reduced  gn- 
inaccessible  table  set,  and  WEELB(M’)  is  that  for  the  weak  EE 
compatibility  relation  on  M 1 .  Thus  Aho  and  Ullman *s  reduced  £?- 
inaccessible  tables  are  included  in  the  more  general  class  of 

weak  SEE  state  compatible  tables. 

It  is  apparent  that  the  technique  of  generating  p  entries  in 
Aho  and  Ullman* s  version  of  Knuth’s  tables  may  be  employed  to 
generate  0  entries  in  the  error  function  E.  Thus  E{j,x)=£i  if  and 
only  if  T{j,x)=0  by  Aho  and  Ullman*  s  method  {¥j,  ¥x)  .  The  error 
function  may  thus  be  made  into  an  incompletely  specified  boolean 
function. 

In  the  next  chapter,  the  use  of  modified  EFA  tables  that 
result  from  DeBemer*s  algorithm  will  require  that  Aho  and 

Ullman* s  first  algorithm  be  modified  to  generate  correct  $- 
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inaccessible  tables.  The  use  of  SEE  states  or  DC  states  in 
compatibles  does  not  reguire  the  development  or  modification  of 

s 

an  algorithm  for  these  modified  tables.  Thus  the  definitions  of 
weak  SEE  and  DC  are  less  dependent  upon  the  type  of  table 
structure  than  the  definition  of  weak  EE. 

Recovery  of  the  EFA  Structure 


Only 

the  terminal 

symbol  portion  cf 

the  error  fu 

net  ion 

be  retained  since  error 

detection  is  perf 

ormed  only  on 

acces 

terminal 

symbols.  If 

an  application 

required  the 

re ad in 

input  stream  non- ter mina 1  symbols,  the  entire  error  func 
would  be  required.  (Typical  applications  could  inc 

incremental  parsing,  context-sensitive  parsing,  etc.,  u 
factored  error  detection  and  a  reduced  matrix  structure  for 
FSM  controller.)  We  observe  that  maintaining  the  entire  e 
function  intact,  and  using  the  DC  state  definition,  the  orig 
structure  of  the  machine  M  may  be  recovered  from  the  fact 
reduced  EFA  (M",E).  Maintaining  only  the  terminal  symbol  por 
dees  not  permit  this.  Osing  the  SEE  state  ccmpatibi 
definition,  the  original  machine  structure  rnay  be  recovere 
the  extent  of  error  equivalence  of  the  restructured  machine 
( M  n  ,  E )  . 

FSM  State  Reduction 


need 
sing 
g  of 
tion 
lude 
sing 
the 
r  ror 
ina  1 
ored 
tion 
lity 
d  to 
with 


The 
re  duced 


preceeding  definitions  simplify 
error— egui  valent  EFA  (Mn  ,E)  from  M 


the 


generation  of  a 
E)  .  A  suitable 
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enumerative  generation  process  may  be  stated  that  is  simplier 
than  the  usual  Paull-Onger  technique: 

a)  Form  the  set  of  maximal  DC  state  blocks  of  M*. 

b)  Select  from  this  set  a  subset  that  covers  the 
state  set  S  of  M’,  such  that  the  number  of 
selected  blocks  is  minimal. 

Had  the  usual  Paull-Unger  process  been  employed  using  the  SEE 
ccmpa tibility  relation  between  states,  the  selected  subset  would 
also  be  constrained  to  be  closed.  But,  as  shewn  previously,  all 
covers  of  DC  blocks  are  closed. 

FSM  Symbol  Reduction 

The  number  of  symbols  belonging  to  a  grammar  vocabulary  may 
not  readily  be  reduced  in  number  since  they  have  been  assigned  by 
the  language.  However,  the  effective  number  of  symbols  may  be 
decreased  by  reducing  the  FSM  M *  by  symbols  {ie-  by  columns)  in  a 
manner  analagous  to  state  reduction.  If  M*  were  transposed, 
reduced  by  the  abeve  reduction  procedure,  then  transposed  again, 
the  effect  would  be  the  same  as  a  column  reduction.  Thus  all  the 
preceeding  results  concerning  states  may  be  applied  to  columns. 

Luccio  [1966]  has  pointed  out  that  the  merging  of  a 
compatibility  class  of  columns  in  an  ISFSM  is  dependent  upon  the 
c crapa t ibil ity  class  cf  rows  of  the  machine.  We  observe  that  the 
use  of  DC  as  the  column  compatibility  relation  eliminates  this 
dependency.  Furthermore  because  of  the  structure  of  the  LR{k) 
FSM  recognizer  M*  of  EFA  M,  we  observe  that  DC  is  the  existing 
compatibility  relation  between  columns  of  M 1 . 
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S i mill taneou s  State-Symbol  PSK  Reduction 


Grasselli  and  Luccio  [1966]  have  shown  that  simultaneous  row- 
column  (state-symbol)  reduction  of  FSM  tables  may  produce  a  more 
minimal  machine  than  either  row  reduction  cr  column  reduction 
alone.  Their  technique  for  simultaneous  reduction  is  an  adaption 
cf  their  state  reduction  technique  [Grasselli  and  luccio  1965] 
applied  to  two  dimensions.  We  seek  a  simplification  of  their 
process . 

The  joint  presence  of  a  set  of  rows  and  a  set  of  columns  in  a 
generalized  reduced  cover  places  an  additional  constraint  on  the 
reduction  process,  termed  _joj.nt  closure  by  Grasselli  and  Luccio. 
This  constraint  arises  because  the  common  table  entries  of  the 
rows  and  columns  to  be  reduced  must  be  merged  into  a  single  cell 
of  the  reduced  generalized  cover..  Thus  if  states  j  and  k  €  5  and 
symbols  x  and  y  6  I  were  to  be  merged,  joint  closure  must  be 
guaranteed  among  entries 

F{j,x),  F(j,y),  F(k,x),  F(k,y)  . 

This  implies  that 

a)  All  output  entries  for  the  above  must  be  identical 
whenever  specified; 

b)  All  transition  entries  for  the  above  must 
constitute  a  compatibility  class  of  rows,  say  W; 

c)  R,  or  a  row  compatibility  class  containing  B,  must 
be  closed  with  the  selected  covering  set  of  row 
compatibles. 
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If  a  set  of  compatible  rows  R  and  a  set  cf  compatible  col urans 
C  of  a  FS  M  M *  satisfy  the  above  conditions-  a)  and  b)  ,  they  are 
said  to  constitute  a  select to nab le  couple  {an  s-couple) . 


Using  the  above,  Grasselli  and  Luccio  have  stated  the 
following  theorem:  a  set  of  row  compatibles  E  and  a  set  of 
column  compatibles  C  form  a  generalized  cover  of  M 1  if  and  only 
if 


D 


2) 


3) 


4) 


Every  state  s  in  S  in  included  in  at  least  one  state 
compatible  of  R; 

Every  symbol  x  in  I  is  included  in  at  least  one 
symbol  compatible  of  C; 

Every  state  compatible  in  R  forms  an  s-couple  with 
every  symbol  compatible  in  C; 

R  and  C  are  closed  (ie  -  satisfy  row  closure,  column 
closure,  and  joint  closure) . 


It  is  clear  that  SEE  may  be  used  as  the  definition  of 
compatibility  in  Grasselli  and  Luccio’s  theorem  to  enable  the 


generation 


of 


minimal 


error-detecting  reduced  EFA's. 


Furthermore,  because  of  the  observation  that 


DC 


is 


the 


compatibility  relation  between  columns  and  between  rows  of  LALR 
grammar  EFA  components  iV  ,  the  row  and  column  closure  constraints 
are  eliminated.  It  is  also  clear  that  if  the  DC  definition  were 
used  instead  of  SEE  compatibility  in  a  general  EE A  that 
simplifications  result  by  eliminating  the  row  (and  column) 
closure  restrictions.  In  addition,  the  s-couple  definition  may 
be  rephrased: 
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Definition _ 7:  If  j  and  k  6  S  are  any  two  states  of  M1  in  state 

DC  class  R,  and  x  and  y  G  I  are  any  two  symbols  of  H*  in 
symbol  DC  class  C  such  that 

F<j,x)  -  F  ( j,  y)  -  F  (k,x)  -  F  (k,y) 
for  any  of  these  which  are  specified,  then  R  and  C 
constitute  a  DC  s-couple,  where  F^ff1,!2}, 

a 

Clearly,  DC  s-couples  have  singletcn  state  successors 
f i  (j,x) ,  for  all  x  in  I,  forming  a  singletcn  state  compatibility 
class.  Since  every  state  in  S  is  covered  by  some  state  DC  block 
in  R,  the  singleton  state  successor  of  R  is  covered;  hence  it  is 
also  closed.  This  eliminates  the  joint  closure  requirement  in  a 
generalized  cover  of  DC  s-couples.  Thus  analagous  to  Grasselli 
and  Luccio’s  theorem,  we  may  state  the  following  without  proof: 

Theorem _ 6:  A  set  of  DC  state  blocks  R  and  a  set  of  DC  symbol 

blocks  C  form  a  generalized  cover  of  M *  if  and  only  if 

1)  Every  state  is  included  in  at  least  one 

DC  state  block  of  B; 

2)  Every  symbol  is  included  in  at  least  one 
DC  symbol  block  of  C; 

3)  Every  state  DC  block  R  forms  a  DC 
s-couple  with  every  DC  symbol  block  in  C.  » 


A  technique  for  the  generation  of  generalized  DC  covers  may 
now  be  considered  based  on  the  notion  of  DC  "tuples”. 


Definition _ 8:  A  tuple  "t"  of  FSd  d»  is  a  set  {j,x} ,  where  jGS 

and  X6I. 
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initio n_9:  A  tuple  t={j,x]  and  a  tuple  t,=  {k,y}  are  DC  if  and 

only  if 

1)  j  B  k, 

2)  x  §  y, 

3)  states  j ,  k  and  symbols  x,  y  form  a  DC 
s-couple . 

If  tuples  t  and  t1  are  not  DC  then  they  are  disjoint 
incompatible.  a 

The  technique  for  simultaneous  row-column  reduction  is  thus 

a)  Form  the  set  of  maximal  DC  tuple  blocks  of  M*. 

b)  Select  from  this  set  a  subset  that  covers  all 
states  j  €  S  and  all  symbols  x  €  I  such  that 
the  number  of  selected  subset  blocks  is 
minimal . 

The  effectiveness  of  the  reduction  process  may  be  determined 
from  the  DCLBfM1),  using  the  tuple  DC  relationship  rather  than 
the  state  DC  relationship  in  the  computation.  As  was  the  case 
for  the  lower  state  bound,  it  may  be  shown  also  for  tuples  that 

GLB  (M 1 )  <  S  EELB  (ft  1 )  <  DCLE  {  M 1 ) 

and  for  a  DeRemer  LALR  EFA 

GLB  (II1)  <  SEELE(H»)  =  DCLB  (M  * )  . 

We  defer  to  the  next  chapter  the  practical  problems  of 
generating  reduced  error- detecting  EFA's. 
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State  Assignment 


It  is  well  known  [Wood  1968]  that  the  proper  choice  of  a 
state  assignment,  for  a  FSM  can  reduce  the  complexity  of  its 
realization.  However,  the  selection  of  a  suitable  assignment  is 
non-trivial,  and  heuristic  techniques  must  be  employed. 

The  state  assignment  we  adopt  is  founded  on  a  base- 
displacement  (ED)  relative  addressing  scheme  for  encoding  FSM 
transition  entries  f1.  By  properly  selecting  the  assignment,  the 
base  components  may  be  factored  from  the  FSM  matrix  M*,  leaving 
the  displacement  components.  The  effect  of  this  assignment  is 
two-fold : 

1)  The  data  width  (number  of  bits)  needed  to  store 
transition  information  is  reduced,  since  the 
displacement  ranges  typically  are  a  small  fraction 
of  the  absolute  state  transition  number; 

2)  More  similarity  among  states  ard  among  symbols  may 
result  in  the  assigned.  JJ  *  than  in  the  unassigned  M*. 

If  greater  commonality  among  states  and  among  symbols  can  be 
achieved  by  an  assignment  that  exploits  the  inherent  structure  of 
the  FSM  component  M*  of  the  EFA  recognizer,  then  the  reassigned- 
then-reduced  FSM  M"  may  contain  fewer  compatibility  classes  in  a 
generalized  cover  than  an  unassigned— then-reduced  FSM. 

An  optimal  assignment  would  enable  all  the  inherent  FSM 
structure  to  be  factored  from  M1  into  the  base  address  component, 
leaving  minimal  data  width  displacement  entries,  while  maximizing 
commonality  among  states  and  symbols  of  M*.  This  would  permit 
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the  generation  of  a  maximally  reduced  MM.  We  leave  to  the  next 


chapter  the  practical  aspects  of 


a  suitable 


sub-opt imal 


assignment . 


■ 


•* 


' 


•- 


' 


. 

. 

' 

' 
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Chapter  3 


Practice 

Practical  Implications 

As  mentioned  at  the  beginning  of  the  last  chapter,  we  are 
interested  in  the  EFA  part  of  DP DA  recognizers  of  LR  (k)  grammars. 
Our  goal  has  been  to  produce  practical  parsers  for  these  grammars 
whose  tabular  EFA’s  are  as  small  as  possible,  without  sacrificing 
the  speed  of  parser  operation  or  the  ability  of  the  parser  to 
detect  the  same  input  errors  at  the  same  parse  step  as  the 
unreduced  EFA’s.  Our  approach  to  this  problem  has  been  by  means 
oE  FSM  minimization  theory  adapted  for  the  production  of  correct 
error-detecting  parser  tables. 

The  implications  of  our  approach  are  straightforward:  if  an 
EFA  may  be  formally  minimized,  its  tabular  representation  will  be 
as  small  as  practical.  In  addition,  by  preserving  a  matrix 


struc 

ture  in  a 

minimized  EFA, 

a  direct  matrix 

indexing 

technique 

may 

be  used 

to  access  the 

parser  tables. 

This  combination  of 

fast 

table  access,  implying 

fast  parsing. 

and  small 

parsing 

tables  makes  our  approach  attractive. 

From  a  theoretical  standpoint,  the  problems  of  generating 
formally  minimized  EFA  tables  for  an  LR  (k)  grammar  have  been 
solved.  The  SEE  definition  provides  a  condition  for  the 
production  of  correct  error-detecting  compatible  blocks  of  the 
error-factored  FSM  H*,  and  the  Paull-Onger  covering-closure 
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algorithm  provides  a  technique  f 
The  DC  definition  simplifies 
generation  of  error-detecting  DC 
always  closed.  Thus  a  simpli 
used  in  this  case. 


or  generating  a  reduced  F5M  NM. 
this  process  by  providing  for  the 
blocks  of  M*  whose  coverings  are 
fied  Paull-IInger  algorithm  may  be 


In  this  section,  three  algorithms  are  presented  for  the 
minimization  of  the  FSM  component  M’  of  the  error-factored  EFA 
{ M  * ,  E) .  All  three  are  approximate  in  nature  (ie-  they  are  not 
formal  in  the  sense  of  the  formal  Paull-Unger  algorithm) ,  but 
rather  they  generate  good  approximations  to  a  theoretically 
minimal  reduced  cover  of  M * .  As  will  be  demonstrated  in  the  next 
chapter,  the  algorithms  often  produce  theoretically  minimal 
solutions  for  . 


We  will  consider  a  practical  state  assign 
class  of  LR(1)  grammar  EFA*s,  followed  by 
semanticless  chain  productions  from  the 
concludes  with  a  discussion  of  the  measures  o 


reduced  EFAfs. 


ment  algorithm  for  a 
the  elimination  of 
EFA.  The  chapter 
f  effectiveness  of 
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f 

t 


Algorithm  #1  re 
elution  of  the  state 
roia  EFA  W=(M*,E)  o 
hat  Algorithms  #2  an 


presents  an  initial  appro 
minimization  problem  for 
f  an  LR  (k)  grammar.  It  is 
d  #3  are  derived  from  it. 


ach  to  a  pract 
FSM  M 1  der 
important  onl 
The  algorithm 


ical 
ived 
y  in 
is : 


a) 


Using  the  no 
maximal  DC  sta 
method  of  Roy 


ticn 

of 

state 

te 

blocks 

of 

an  d 

Sheng  [  1972 

pair 
M  *  by 

]• 


DC,  form  the  set  of 
the  decomposition 


b)  Select  from  this  set  of  blocks  a  subset  of  blocks  that 
covers  all  states  j  in  S,  such  that  the  number  of  subset 
blocks  is  minimal.  Use  the  approximate  method  of  Bowman 


and  McVey  [1970]. 


It  is  evident  that  Algorithm  #1  is  essentially  a  version  of 
the  Paull-Unger  technique,  using  the  definition  of  DC  so  that 
there  are  no  closure  constraints  on  the  solution.  The  technique 
of  Roy  and  Sheng,  and  that  of  Bowman  and  McVey  were  chosen  for 
their  purported  simplicity  and  for  their  execution  efficiency 
when  implemented  in  a  computer  program.  An  exposition  of  Roy  and 
Sheng’s  method  is  given  in  Appendix  VIII. 


Algorithm  #2 


Algorithm 

#2  is  a  refinement 

of  Algorithm 

*1, 

and 

is  based 

upon  empirical 

observations  of  its 

operation . 

It 

was 

not  iced 

that  the  state  decomposition  seq 

uence  developed 

by 

Roy  and 
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Sheng's  method  in  the  production  of  the  set  cf  maximal  DC  blocks 
closely  resembled  the  state  composition  sequence  followed  in 
Bowman  and  McVey's  method.  This  similarity  has  been  exploited. 

The  algorithm  produces  a  set  of  cover  blocks  B={B(k)}  that 
forms  a  state  set  partition.  The  intersection  of  block  B{i)  with 
E  (j)  is  (¥i^j  in  k) .  It  uses  the  decomposition  process  of  Roy 
and  Sheng  combined  with  a  simple  criteria  for  choosing  a  locally 
optimal  compatible  block  after  each  major  decomposition  step. 
The  algorithm  follows. 

Algorithm  #2 

Step  1)  Form  the  set  of  pairwise  disjoint  incompatible  states 
D  (s)  for  each  state  cf  M*  (1<s<jN'|). 

Step  2)  Determine  the  state  decomposition  sequence  SEQ  by  the 
distribution  table  method  of  Roy  and  Sheng,  where  SEQ(i) 
is  the  i-th  state  in  the  deccm pos ition  sequence,  and 
1<i<#steps,  where  #s teps= 1 SEQ | .  (See  Appendix  VIII.) 

Clear  the  count  of  the  number  of  blocks  in  the  cover  set 
by  setting  fblocks  =  0. 

Clear  the  outer  loop  index:  1=0. 

Step  3)  Set  1=1+  1.  If  I  >  #steps,  finish. 

Set  s  =  SEQ (I) . 
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Step  4) 


If  state  s  has  been  included  in  {E  (k)  ,  ¥  k<#blocks}  then 
go  to  step  9)  .  (Note  that  3  (k)  is  the  k-th  selected 
cover  block.) 


Step  5)  Form  the  set  of  states  that  are  pairwise  DC  with  state 
s:  ie-  form  the  set  complement  of  D (s)  ,  called  D  (s) . 

Delete  from  D(s)  any  states  included  in  {E  (k)  }  ,  ¥ 
1<k<#blocks.  This  set  R ( 1 )  forms  the  initial  set  of 
potentially  maximal  DC  states:  ie-  form  3(1)  by  the  set 
difference 

3(1)  =  D(s)  -  {E  ( k)  ,  ¥  k<#blocks}  . 

Initialise  the  inner  loop  index:  J  =  I. 

Step  6)  Set  J  -  J  +  1 .  If  J  >  #steps,  go  to  step  8) . 

Set  s*  =  SEQ  (J)  . 

Step  7)  If  state  s’  is  contained  in  3(g)  for  some  1<g<|Rj,  and 
if  seme  of  the  pairwise  disjoint  incompatible  states  of 
s’  are  also  contained  in  3(g)  then  form  two  residue  set 
blocks  E1(g)  and  32  (g)  by  set  difference  such  that 
E 1  (g)  =  3(g)  -  {s’} 

R2  (g)  =  3  (g)  -  D  (s * )  . 

Otherwise 

R1  (g)  =  R  (q) 

R2  (q)  =  JJ. 

Repeat  step  7)  above  for  all  1<g<J31,  then  loop  back  to 
step  6)  . 
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Step  8) 


Replace  P  (q)  by  R1  (g)  and  R2  (q)  ,  then  set  R  1  (q)  =  R2  (g )  =0 , 
where  1<q<  j  R 1 j  . 

Choose  a  locally  "Eaxiaal"  block  R ( p)  such  that 
HUp)  I  >  1 H  (q)  1  ,  where  1<q<]H  | 
ie-  a  block  with  the  greatest  number  of  states.  If 
there  is  more  than  one  such  block, choose  the  first  cne. 

Set  tblocks  —  tblocks  *•  1.  Add  the  locally  maximal 

block  to  the  set  of  cover  blocks:  E  (^blocks)  =  R  (p) . 

Step  9)  If  the  set  of  cover  blocks  {B(k}}  does  not  cover  all 
states  of  the  flew  table:ie 

1  {B  (k)  ,  ¥  1<k<#blocks}  1  <  |  H *  | 
then  loop  back  tc  step  3) . 

Step  10)  Finish.  a 

Algorithm  #2  uses  only  the  states  that  are  pairwise  DC  with 
state  ”s”  not  covered  by  a  cover  block  chosen  at  a  previous 
decomposition  step  as  the  initial  potentially  maximal  DC  state 
block.  This  avoids  the  generation  of  superfluous  state  DC 
compatibles  by  using  the  smallest  possible  initial  state  set 
B{1),  and  using  only  the  decomposition  relations  that  are 
applicable  to  it. 
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Algorithm  # 3 


A  third  approximate  algorithm  may  be  formed,  based  on  an 
implementation  refinement  of  Algorithm  #2.  Algorithm  #3  improves 
on  Algorithm  #2  by  eliminating  the  enumeration  of  the  set  of 
potentially  maximal  DC  blocks,  while  producing  the  same  cover  set 
as  Algorithm  #2. 

In  step  7  cf  Algorithm  #2,  the  set  R1  (q)  is  formed  by 
deleting  state  s’  from  R  (g)  .  Thus  R{g)  contains  at  most  one  more 
state  than  R1{g) •  It  is  possible  to  ensure  after  the  set 
replacement  of  R  by  R1  and  R2  in  step  8  that  element  R(1)  is  the 
block 

R(1)  =  8i(1)  ~  {s’}  , 
or  equivalently  R(1)  =  R(1)  -  {s’}. 

If  the  first  part  of  step  7  is  executed,  then  the  following 
is  true: 

|R2  (q)  1  >  1. 

But  this  implies  that  at  each  decomposition,  R1 (g)  will  contain 
at  least  the  same  number  of  states  as  R2  (g)  ,  thus 

I  R 1  (q)  I  >  1  R2  (g)  I  • 

But  element  R  (g)  ,  for  g>1  has  been  formed  from  seme  element  R2(g) 
in  a  previous  decomposition  step.  Therefore 

1R(1)  |  >  Jfi(q)  j,  ¥  1  <  q  <  ]  R  |  . 

But  the  criteria  for  selecting  the  locally  optimal  compatible  in 
step  8  will  select  R(1)  from  (R(g)}.  Thus  the  enumeration  of  the 
sets  R 1  and  R2  may  be  replaced  by  a  non-enumera t i ve  process  that 
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deletes  state  s*  from  the  initial  block  R(1)  if  and  only  if  s'  is 
contained  in  H(1)  and  R(1)-D{sl)  is  not  empty. 

Algorithm  #3  may  be  formed  from  Algorithm  #2  fcy  replacing 
steps  7  to  10  by  the  following  steps: 

Step  7)  If  {s'}  c  R  (  1) 

and  B  (1)  -C(s')  *  then 

R(1)  =  B  ( 1)  -  {s1}  . 

Loop  tack  to  step  6)  ♦ 

Step  8)  Set  ^blocks  =  #blocks  *  1. 

Set  E  (#blocks)  =  B  (1)  . 

Step  9)  If  1  [E  (k)  ,  ¥  1<k<#blocks} J  <  jM'j,  then  loop  tack 
to  step  3)  . 

Step  10)  Finish.  „ 

Algorithm  Complexities 

The  efficiency  of  the  reduction  algorithms  may  be  estimated 
from  an  analysis  of  their  respective  complexities.  Let  M ’  be  an 
error-factored  FSK  of  EFA  having  MnM  internal  states.  Let 

the  number  of  decomposition  steps,  as  determined  from  the 
technique  of  Boy  and  Sheng,  be  "p".  In  general  it  may  be  shown 
that 

p  <  min  (k-  1 ,  n-  1) 
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where 

k 

=  the 

number 

of  unique 

pairwise 

inco mpa  tibi lity 

r elat io 

ns. 

Assume 

that  the 

complexity 

measure 

desired  is  the 

number 

of 

(maxima 

1) 

blocks 

generated 

from  which 

the  set  of  cover 

blocks 

is 

selected 

• 

Complexity  of  Algo r it h m  # J. 

Start  the  decomposition  process  with  one  block  composed 
of  n  states.  Each  step  in  the  decomposition  process 
generates  a  maximum  of  two  blocks  from  each  block  generated 
in  a  previous  step.  Therefore  after  the  i-th  step,  a  maximum 
of  2**1  blocks  will  have  been  generated.  Thus  the  complexity 
of  Algorithm  #1  is  0(2**p).  In  the  worst  case,  the  value  of 
p  is  approximately  egual  to  that  of  n,  hence  the  complexity 
of  Algorithm  #1  is  0(2**n). 

Complex it y  of  Alq or it h m  # 2 


Let 

{si , s2,  .  .  . 

r  S  p] 

be  the  state 

s  In 

the  decom 

position 

sequence 

of  "p”  steps. 

as 

determined 

by 

Roy  and 

She  ng  1  s 

method . 

Let  {D  1 ,  D 2  , 

•  •  • 

Dp)  be  the 

set 

of  states 

that  are 

pairwise  compatible  with  [s1,s2,  ...  ,sp}  r es pect ively :  ie  - 
state  si  is  pairwise  compatible  with  the  states  in  Cl,  etc. 
(The  number  of  states  in  Ci,  for  any  i,  is  less  than  or  equal 
to  n,  by  definition.) 


Let  {B 1 ,  E  2 ,  ...  ,Br]  be  the  set  of  cover  blocks  selected 
by  Algorithm  #2,  where  r  <  n. 


. 

■ 


The  following  result  will  prove  useful:  for  a  binary 
decomposition  process,  if  there  are  "a"  states  to  be 
decomposed,  and  there  are  ”bn  decomposition  steps,  then  the 
maximum  number  of  unique  blocks  generated  in  the 
decomposition  process  may  be  shown  to  be  2**min(a,b)  . 

For  the  first  cover  block,  decompose  by  state  si:  select 
D 1  as  the  initial  maximal  block.  Perform  a  maximum  of  (p-1) 
decomposition  steps,  producing  a  maximum  of 
T  1  =  2**min  { j  E  1  |  ,  p-  1) 

unique  blocks.  From  these,  select  the  largest  cover  block  B1 
by  step  8)  of  Algorithm  #2. 

For  the  second  cover  block,  decompose  by  state  s2: 
select  D2,  and  delete  from  it  those  states  covered  by  El. 
(Form  the  set  difference  of  the  sets  D2  and  El,  This  is 
represented  here  as  D2  -  El.)  Perform  a  maximum  of  (p-2) 
decompositions  on  D2  -  El,  producing  a  craximum  of 
T 2  =  2**min  { | D2-E1 ] ,p-2) 

unique  blocks.  From  these,  select  the  largest  cover  block  B2 
as  above. 

In  general,  for  the  i-th  cover  block  (1<i<r) ,  decompose 


by  state 

si:  select  Di  and  delete 

from 

this 

/ 

set  those 

states 

included 

in  the  cover  blocks  Ej 

(¥  j< 

i)  • 

Represent 

this  as 

the  set 

Di-Bi*,  where  Ei*  is  the 

set 

unio 

n  of  E  j 

(*j<i)  . 

Now  per 

form  a  maximum  of 

(P~i) 

decomposition 

steps. 

generating  a  maximum  of 

Ti  =  2**min  { | Di-Ei * ! , p-i) 
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unique  blocks. 


The  total  number  of  blocks  generated , from  which  cover 
blocks  are  selected,  is 

T  -  T1  +  T2  +  ...  +  T  r 
=  SUd  (Ti)  ,  for  1<i<r 
-  S0M2**min(IDi-Bi»  |,P“i)  )  ,  for  1<i<r. 

A  least  upper  bound  for  this  series  may  be  calculated 
using  (p-i)  as  the  exponent.  In  this  case 
T  =  SUH  {2**  (p-i) )  ,  for  1  <i<r 


0  (2**p)  . 

Thus  the  least  upper 

bound  on 

complexity  is 

0(2* *p)  ,  or 

approximately 

Algorithm  #1. 

0 (2**n) , 

which  Is 

the  same  as 

that  for 

If  the  states  in  Bi1  overlap  with  those  of  Bi,  such  that 
JDi  -  Bi*  |  <  p-i 

then  the  complexity  of  the  exponent  of  each  term  of  T  will  be 
reduced.  Thus  the  order  of  complexity  of  Algorithm  #2  will 
be  less  than  that  of  Algorithm  #1,  but  still  combinatorial. 

Complex it v  o f  Algo r ithm _ £ 3 

Since  Algorithm  #3  performs  no  intermediate  enumeration 
during  the  generation  of  B,  the  space  complexity  is  simply 
the  number  of  initial  blocks  R(1),  or  JBJ.  In  the  worst 
case,  this  is  the  number  of  decomposition  steps  p.  Thus 
Algorithm  #3  has  a  linear  space  complexity  of  p,  or 
approximately  0  (n)  . 
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The  analysis  of  the  two  nested  Iccps  of  the  algorithm 
shows  that  the  time  complexity  is  0  (p2) ,  or  approximately 
0(n*). 

Algorithm  #1  takes  advantage  of  the  DC  condition  in  the 
determination  of  the  compatibles,  and  takes  advantage  of  the 
decomposition  method  of  Eoy  and  Sheng  and  the  covering  method  of 
Bowman  and  McVey  to  implement  a  version  of  the  Paull-Unger 
reduction  technigue.  Algorithms  #2  and  #3  are  implementation 
improvements  of  Algorithm  #1.  From  the  rudimentary  complexity 
analysis,  it  is  clear  that  Algorithm  #3  should  be  superior  to 
both  Algorithm  #1  and  Algorithm  #2. 

Algorithm  #3  Effectiveness 


T 

he 

space 

a 

nd  time 

complexities 

of 

Algorithm  # 

provi 

de 

a  ineas 

ure 

of  the 

minimality 

of 

th 

e  generated 

Becau 

se 

the 

alg 

orit h  ra 

is  approximate. 

f  c 

rmal  proofs 

minim 

ali 

ty  are 

not  possib 

le  for  a  general 

M*  . 

Even  a  ” 

FSM  M 

•  i 

s  not 

rea 

dily  fou 

nd  for  the  class 

of 

LR  (k)  gramma 

it  is 

un 

known 

if 

it  is 

theoretically 

poss 

ible  to  sh 

Algor 

ith 

m  #3 

produces  a 

minimal  or  a 

ne 

ar-  c 

inimal  reduc 

£3 11  Of 

M 1 

.  In 

pra 

ctical  circumstances. 

th 

e  st 

ate  DC  lows 

may 

be 

calcu 

lat 

ed  and 

compared  to  1 

M" 

1  /  a 

nd  thus  the 

solut 

ion 

minim 

ali 

ty  deter 

rained . 

W 

e 

expect 

A 

Igor ithm 

#3  to  prod 

uc 

e  »' 

reasonably11 

solution  FSK 1 s  MM.  As  mentioned  in  the  last  chapter,  the 
structure  of  ft*  is  such  that  only  a  limited  number  of  different 
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state 
sy  mho 
any  o 
usage 
res  tr 
every 
the  ru 
be  e 
state 
cover 


transitions  {f1{k,x)r  -VkCS]  may  app 
1  column  ”x" .  These  state  transitio 
ther  symbol  column  "y"  (y#x)  .  Th 

of  grammar  vocabulary  symbols 
icted  set  {f  1  (k,  x)  }  )  ,  in  the  grammar 
state  will  have  transitions  unde 
ajority  of  entries  in  PS^’s  of  large 
mpty.  Thus  there  is  a  reasonabl 
s  j  and  k  being  pairwise  DC  and  thus 
block . 


ear  in  the  F S M  under  any 
ns  will  not  appear  under 
e  localized  restricted 
(which  gives  rise  to  the 


BNF  suggests 

t  hat 

not 

r  all  symbols. 

in  f 

act , 

grammars  will 

prob 

ably 

e  possibility 

of  any 

two 

mergeable  into  a  DC 


If  Algorithm  #3  provides  an 
a”,  it  is  because  Algorithm  #2 
provides  a  near- mini mal  solution 
multiplicity  of  equally  good 
decomposition  step.  Because  an 
set  of  "locally  heaviest"  blocks 
reasonable  that  a  multiplicity 
near-minimal  solutions  ft"  exist. 
Algorithm  #3  is  one  of  these. 


effective  nea 

r-  min 

iroal  solution 

also  does. 

If 

Algorithm  #2 

,  it  is  due  to 

the 

presence  of  a 

CC  blocks  Ei 

at 

every  major 

arbitrary  selection  of  one  of  a 
has  been  made  for  Ei,  it  is 
of  equally  acceptable  minimal  or 
The  particular  M"  found  by 


Even  if  a  large  number  of  equally  minimal  solutions  did  not 
exist,  the  decomposition  procedure  of  Algorithm  #2  is  effective 
in  generating  only  the  maximal  DC  blocks  reguired  by  using  a  near 
optimal  decomposition  sequence.  The  sequence  is  such  that  states 
having  the  least  number  of  pairwise  DC  states  are  decomposed 
prior  to  those  having  a  greater  number.  Thus  the  first  major 
decomposition  step  will  generate  a  cover  block  for  the  most 
difficult  to  cover  state.  Included  in  this  block  will  be  several 
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states  that  are  relatively  simple  to  cover  because  they  have  many 
pairwise  states  with  which  they  are  DC.  Thus  the  enumeration 
required  for  block  B1  produces  a  set  of  DC  compatibles,  most  of 
which  are  equally  acceptable  in  forming  a  partial  covering  of  M 1 . 
The  second  block  E2  will  cover  the  second  most  difficult  to  cover 
state,  along  with  some  other  easily  covered  states,  but  with 
reduced  enumeration.  Since  the  most  easily  covered  states 
produce  the  greatest  enumeration  in  the  decomposition, 
elim  inating  these  from  the  initial  potentially  maximal  block 
prior  to  the  decomposition  procedure  markedly  reduces  their 
combinatorial  proliferation  at  each  decomposition  step. 


The  next  chapter  contains  results 
effectiveness  of  the  algorithms  in  generating 
state  covers  and  symbol  covers  fcr  M  *  for  som 

Simultaneous  State-Symbol  Deduction  by  Alg.-  #3 

Consider  the  application  of  Algorithm 
state-symbol  reduction  of  M*  (by  replacing  ”st 
Algorithm  #3) .  While  the  algorithm  should  per 
produce  reasonably  minimal  DC  tuple  genera 
workspace  required  for  the  formation 
incompatibility  relations  would  be  prohi 
implication  table  technique  [Pauli  and  Unger 
contain  the  pairwise  tuple  relationships, 
implementation  would  require  a  workspace  of  ap 
entries  for  "n"  states  and  f,m”  symbols.  Even 


illustrating  the 
reasonably  minimal 
e  typical  grammars. 


#3 

to 

s imu ltan 

eou  s 

te" 

b 

y  ’’tuple" 

in 

crm 

a 

dequa  tely 

and 

ize 
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.  If 
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were  use 

d  to 
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r aightf or 
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ately  (nm 

)  2/2 

if 

o 

ne  bit 

were 
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the  storage  space  for  even 


used  for  each  implication  table  entry/ 
a  moderate  size  grammar  would  be  excessive 


The  definition  of  DC  tuples  enables  the  elimination  of 
classes  of  implication  table  entries.  For  example,  if  j  and  k  £ 
S  are  not  DC,  then  all  tuples  involving  these  states  may  be 
deleted  from  the  implication  table.  Similarly  if  x  and  y  €  I  are 
not  DC,  all  symbols  involving  these  tuples  may  he  deleted.  Thus 
an  alternate  data  storage  mechanism  could  be  used  for  practical 
purposes,  perhaps  some  sort  of  list  structure.  Thus  the 
practical  problems  of  simultaneous  state-symbol  reduction  remain 
u  nsol ved . 


ETC  and  CTR  Reduction 


As  an  alternative  to  simultaneous  row-column  reduction, 
consider  row  reduction  followed  by  column  reduction  on  the 
reduced  row  matrix,  or  column  reduction  followed  by  row  reduction 
on  the  reduced  column  matrix.  These  processes  will  be  called  RTC 
{row- then-column)  and  CTR  (ccluran-then-r o w)  reduction 
respectively . 

The  implementation  of  the  RTC  and  CTR  processes  is  not 
constrained  by  the  problems  that  hampered  simultaneous  row-column 
reduction.  Algorithm  #3  may  be  applied  twice  in  a 
straightforward  manner  to  produce  the  desired  reduction  process. 
While  the  theoretical  lower  bounds  on  the  sizes  of  the  parser 
tables  produced  by  RTC  or  CTR  reductions  probably  will  not  be  as 
low  as  those  for  simultaneous  rcw-column  reduction,  the  RTC  and 
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CTR 

re 

due 

t  ion 

processes 

are 

actually 

a  form 

of 

con 

s  t  r  a  i  n  e 

a  t 

uple 

decomp 

osi 

t  ion 

.  The  deccmpositicr  s 

eg uence 

is 

det 

er mined 

for 

the 

state 

set 

by 

using  a  co 

ns  tan 

t  value 

for  the 

sym 

bcl 

comoo 

i. 

nent 

of 

the 

t 

upl 

e; 

that  for 

the 

column 

set  is 

f  oun 

d  u 

sing  a 

cons 

tant 

state 

co  m 

pone 

nt  in  the 

state 

-symbol 

tuple. 

Th 

e 

next 

chapter 

conta 

ins  some 

reduct 

ion 

res 

ults  of 

RTC 

and 

CTR  reduction  processes  for  typical  grammars. 

LR  fk)  EFA  Modifications 

We  have  considered  that  the  EFA  M  has  been  obtained  from  the 
LP  (k)  generation  techniques  of  Knuth,  or  equivalently  those  of 
DeRep.Gr.  Thus  all  prior  results  have  related  to  these  table 
structures . 

Following  Anderson  [1972],  we  modify  the  table  structures  by 
removing  all  LR  (0)  reduce  states  from  the  EFA:  ie-  all  states 
containing  only  reduce  entries.  These  states  serve  only  to 
verify  that  certain  input  symbols  are  present  prior  to  performing 
a  reduction  on  the  DPDA  stack  top.  Thus  error  detection  would  be 
delayed  until  after  the  reduction  is  performed.  However,  the 
presence  of  an  invalid  input  symbol  would  still  be  detected  by 
the  error  function  E  on  a  subsequent  symbol  access. 

The  error  detection  capability  of  the  LR  (C)  reduce  states  may 
be  preserved  in  E  of  the  EFA  (M',E)  even  though  all  LR  (0)  reduce 
states  are  deleted  in  a1.  The  retention  of  E(j,x)  for  all  such 
states  "j”  and  all  terminal  symbols  "x"  ensures  this. 
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such  that  a 


The  machine  states  S  are  then  renumbered, 
transition  to  state  "i"  { 1 <i<#product ions)  implies  that 

production  "i"  is  to  be  applied.  These  states  are  referred  to  as 


"apply 

states",  or  as 

"reduce  states". 

A  state 

II  -j  M 

(#product ions< j<n)  is  a 

"parse  state",  or 

a  " 

non  L  R  (0  ) 

reduce 

state  " . 

In  subsequent  practical  algorithms. 

a 

reference 

to  a 

"state" 

unless  otherwise 

qualified  is  to 

be 

inter  pre  te 

d  as  a 

reference  to  a  parse  state. 
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n  additi 

on  to 

the  LR 

(0)  reduce  s 

tate  e 

lira 

ina 

tion , 

th 

e  no 

tion 

of 
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hybrid  L 

H{0)  - 
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composite  p 
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ed 

by 
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is 

emplo 

yed . 

Each 

specif 

led  FSM  en 
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M  » 
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s  o 
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it ion  f1 

and 

an  outp 

ut.  entry  f2= 

{read. 

loo 
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head} 

i-  w 

i  th 

the 

und 

er 

standing 

tha 
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output  act 

ion  i 
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to 

occu 
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r  to 

following  the  state  transition  f1.  Thus  if 

F  ( j,  x)  =  {f 1  {j,  x)  ,f  2  (j  ,x)  }  ,  where  f2  =  ”readM,  and  f*=k  {where 
1 <k<# product ions) ,  input  symbol  x  is  read,  placed  on  the  pushdown 
stack,  and  production  number  "k"  is  applied  to  the  stack  top.  If 
f 2="look-ahead",  input  symbol  x  is  inspected  only,  not  stacked. 


The  above  modifications  are 
size  of  the  parse  tables  without 
capability  of  the  EFA,  and  beca 
structure.  Modifications  of  thi 
DeRemer  [1969]  and  by  Lalcnd 
practical  parser  table  generator 


important  because  they  reduce  the 
imparing  the  error  detection 


use 

!  they 

simpl 

ify  the  LR(0)-LR 

(1) 

s 

type 

have 

been  employed 

by 

e 

[  1972 

]  in 

i m piemen ta  tion s 

of 

s . 
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Practical  State  Assig  nnien  t 


The 

state  assignment 

procedu  re 

assumes  a  set  of 

hybr  i 

LR(1)  , 

or  equivalent. 

FSM 

tables  of 

M 1 ,  modified 

as  s 

above . 

It  follows 

in 

spirit  a 

proposal  by  And 

er  son 

although  he  did  not  present  a  working  procedure  for  accoin 
the  assignment.  His  primary  intention  was  tc  minimize 
width  required  for  storing  data  in  the  transition  ma 
restricting  the  range  of  numbers  in  a  column  or  ro 
matrix.  Our  intention  is  to  increase  the  similarity  of  r 
of  columns,  and  to  exploit  this  similarity  in  a  reduce 
structure. 


d  LR  (0)  - 
pacified 
[  1972], 
p listing 
the  bit 
trix  by 
w  of  the 
ows  and 
d  matrix 


The  State  Assignment  Procedure 

1)  Sort  the  read  states  of  the  table  such  that  states  having  the 
same  state  access  symbols  have  sequentially  numbered  state 
numbers.  Renumber  all  non-apply  machine  transitions 
accordingly. 


2) 
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3) 


read  state  transition.  Repeat  the  above  for  each 
resetting  the  sequence  number  count  tc  zero  each  ti 
column  scan  is  started. 


column , 
me  a  new 


For  the  read  transitions  to  non-apply  states:  place  the 
absolute  state  number  of  the  first  state  of  a  sequence  of 
states  with  a  common  access  symbol  in  a  vector  containing  one 
entry  for  each  machine  symbol.  This  vector  is  the  base 
component  of  the  ED  address  for  the  read  transitions  to  non- 
apply  states. 

4)  Reassign  all  transit  ions  to  apply  states  by  rows  as  follows: 
scanning  from  left  to  right  of  each  table  cow,  assign  the 
first  apply  production  transition  the  sequence  number  zero. 
For  the  second  and  subsequent  apply  transitions,  if  the 
destination  apply  state  has  already  been  assigned  in  the  row 
substitute  the  assigned  sequence  number  for  the  production 
number  in  the  matrix  entry.  If  the  apply  state  has  not  been 
reassigned,  increment  the  sequence  number  count  by  one,  and 
assign  the  new  sequence  number  to  the  apply  transition. 
Repeat  the  above  for  each  row  of  the  flow  table,  setting  the 
sequence  number  count  to  zero  each  time  a  new  row  scan  is 
started . 

5)  Place  the  absolute  apply  production  numbers  for  each  row  in  a 
sublist  in  the  order  they  were  found,  one  entry  per  unigue 
sequence  number.  Form  a  list  of  production  numbers  from  the 
row  sublists,  containing  one  sublist  for  each  row  of  the  flow 
table.  Form  a  vector  of  pointers  with  each  entry  pointing  to 
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the  address  of  the  absolute  production  number  is  computed  by 
adding  the  sequence  number  stored  in  the  matrix  entry  (the 
displacement)  to  the  value  of  the  base  component  for  the  row 
stored  in  the  pointer  vector.  In  the  former  case,  the  value 
stored  in  the  matrix  entry  is  a  displacement  from  an  absolute 
state  number;  in  the  latter  case  the  value  stored  in  the  matrix 
entry  is  a  displacement  from  a  starting  address  where  the 
absolute  production  numbers  for  a  row  are  stored. 

Chain  Production  Elimination 


The  elimination  of  semanticless  chain  productions  of  length 
one  has  been  proposed  by  Anderson  [1972]  and  by  Aho  and  Oilman 
[1972b]  as  a  means  of  speeding  up  the  operation  of  a  parser  for  a 
grammar.  The  application  of  these  productions  may  be  bypassed  by 
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modifying  the 

parser 

table 

structure 

so  that  transitions 

to  chain 

rule  reduce 

sta  tes 

are 

replaced 

by  transitions 

to  the 

corresponding 

parse 

restart  states 

(ie-  the  states  in 

which  the 

parse  would  have  restarted  after  the  chain  rule  reduction). 


Andersen’s  approach  to  chain  rule  elimination  tends  to 
produce  parser  tables  larger  in  size  than  the  initial  tables  M*, 
whereas  Aho  and  Ullman’s  method  tends  to  produce  smaller  tables. 
Ve  adopt  this  latter  method  for  illustrative  purposes,  since  a 
decrease  in  the  number  of  states  prior  to  state  reduction  implies 
a  potentially  smaller  reduced  state  FSM  Mn.  Chain  elimination 
also  increases  the  commonality  among  symbol  columns.  For 
example,  if  F(j,x)  and  F(j,y)  are  two  table  entries  in  state  j  of 
H*r  where  x,y  €  I,  and  a  chain  rule  y:;=x  is  to  be  eliminated, 
typically  the  entry  at  (j,x)  is  replaced  by  F(j,y)  by  the 


algorithm . 

Thus 

if 

chain  elimination 

is  performed 

prior 

to  the 

modification 

of 

M  » 

(by  assignment. 

reduction. 

etc. 

)  /  a 

poten  tially 

smaller 

reduced  FSM  M,r 

may  result 

than 

would 

otherwise  be  possible. 


Aho  and  Ullman’s  chain  r 
designed  to  process  their  SLR  { 1 ) 
have  modified  their  algorithm 
containing  no  LR  (0)  reduce  states 
in  the  procedure  for  ordering  the 
their  elimination. 
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our  chain-eliminated  DeReraer 


tables . 


To 


chain  elimination  must  be  performed  prior  tc  error  factorization 
of  the  EFA.  In  addition,  a  lock-ahead  error  check  must  be 
performed  prior  to  applying  every  reduction  to  ensure  that  proper 
error  detection  occurs.  Eve  [1973]  has  pointed  out  that  failing 
to  do  so  might  allow  invalid  input  strings  tc  be  accepted  by  the 
EFA.  As  previously  mentioned ,  the  error  detect  ion  capability  of 
the  eliminated  LR  (0)  reduce  states  may  be  retained  by  maintaining 
E  ( j,x)  for  all  LR  (0)  reduce  states  jes,  and  for  all  terminal 
symbols  x€I,  even  though  the  corresponding  f1  and  f2  entries  need 
not  be  present.  The  inspection  of  E{j,y),  where  y  is  the  look¬ 
ahead  input  symbol,  permits  proper  error  checking. 

The  following  chain  rule  deletion  algorithm  will  not 
necessarily  eliminate  all  the  chain  rules  specified  due  to  the 
deletion  of  the  LR(0)  reduce  state  table  entries  prior  to  chain 
elimination.  In  this  regard,  it  is  less  effective  than  either 
Anderson’s  or  Aho  and  Oilman’s  technigues.  Its  prime  use  is  to 
illustrate  the  effect  of  chain  rule  elimination  on  parsing. 

Step  1)  Order  the  set  of  semanticless  chain  productions  tc  be 
eliminated:  if  A::=E,  E::=C,  C ; : =D  are  chain  rules,  the 
processing  order  is  C::=E  first,  then  E::=C  next,  then 
A::=E  (ie  -  process  the  chain  rule  at  the  lowest  level 
in  the  derivation  tree  first,  then  process  the  next 
lowest,  etc)  .  Only  one  chain  rule  per  LHS  symbol  may  be 
eliminated. 

Step  2)  Select  the  (next)  chain  production  to  be  eliminated. 
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Step  3)  Scan  down  the  columns  of  the  flow  table  headed  by  the 
LHS  and  RHS  production  symbols:  if  A;:  =  B  is  the  i-th 
chain  rule,  scan  down  the  columns  headed  by  A  and  by  3. 
Let  f1(T,A)=T1  and  f1{T,E)=T2  for  seme  state  T,  where  f1 
is  the  transition  function. 

Step  4)  If  T2  is  an  L5(0)  reduce  state  (ie:  T2  <  fproductio  ns)  , 
go  to  7)  . 

If  T 1  is  an  L  R  { 0 )  reduce  state  {ie:  T1  .<  #product ions)  , 
and  T2  is  not  {T2  >  #prcduct ions)  then  no  merger  is 
possible,  and  chain  rule  i  cannot  be  eliminated  from  the 
parser  table.  Go  to  9) . 

Step  5)  Scan  across  the  rows  of  T1  and  T2  for  all  input  symbols 

x. 


If  (T2,x)  =  ( T 1 ,  x)  ,  go  to  6). 

If  (T2,x)  is  a  lcok-ahead  transition,  or  is  unspecified, 
go  to  6)  . 

If  (T2,x)  is  a  read  transition  or  an  LR{1)  "raad-then- 
apply”  transition,  and  (T1, x)  is  unspecified,  then  move 
(T2,x)  to  (T 1 ,  x)  .  (If  (T1,x)  is  specified,  then  a  merge 
error  is  indicated,  and  production  i  cannot  be 
eliminated  from  the  table.  In  this  case,  go  to  9) .) 
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Step  6)  Loop  back  to  5) 
inspecte  d . 


until  all  input  symbols  x  have 


Step  7)  Replace  the  transition  f 1  (TfB)  by  T1. 


Step  8) 


Replace  all  transitions  to  state  T2  in  row 
transitions  to  T1. 


Step  9) 


Loop  back  to  3) 
inspected  (for  all 


until  all  column  entries  have 
T  in  the  state  set)  . 


Step  10)  Loop  back  to  2)  until  all  chain  rules  have 
processe  d . 

Step  11)  If  any  read  state  T  (T>#product ions)  is  not  accesse 
any  transition  after  chain  rule  elimination,  delete 
from  the  parser  table  and  renumber  all  affected  st 
and  transitions  accordingly. 


Step  12)  Finish.  * 

If  the  chain  productions  have  been  specified  correctly, 
merge  error  indication  in  step  5  should  net  occur.  This  m 
error  indication  and  that  of  step  4  are  the  consequences  of  u 
Aho  and  Oilman’s  algorithm  on  tables  with  no  LR  (0)  reduce  sta 

The  effects  of  this  chain  rule  elimination  techniqu 
parser  speedup  and  FSH  size  reduction  are  considered  in  the 
chapter. 
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Simpl  ification  of  the  Error  Function 


We  have  previously  considered  the  application  of  Aho  and 
UHman*  s  first  algorithm  [Aho  and  Ullman  1972a]  as  a  means  of 
adding  p  entries  to  the  error  function  E.  Since  all  LR  (0)  reduce 
states  are  eliminated  from  M  when  no  chain  elimination  is 
performed,  no  error  checking  during  look-ahead  can  be  done  by 
these  states.  Consequently  Aho  and  Oilman’s  algorithm  for 
producing  their  p-inaccessible  version  of  Knuth  parser  tables  is 
inapplicable  in  these  instances.  A  suitable  version  could  be 
found  to  add  p  entries  in  E.  Since  we  have  chosen  DC  as  the 
definition  for  the  formation  of  a  reduced  M",  we  leave  the 
production  of  a  suitable  algorithm  for  generating  p-inaccessible 
entries  in  modified  tables  for  future  investigation.  We  consider 
an  alternate  technigue  of  adding  p  entries  to  B  that  does  not 
alter  the  error  detection  capability  of 

The  reduced  machine  M”  produced  by  any  of  the  reduction 
techniques  (state,  symbol,  ETC,  CTR,  etc.)  may  contain  p  entries 
in  its  structure  after  reduction.  Let  the  states  belonging  to 
the  DC  state  block  A  be  (a{i),  0<i<jA|},  and  let  the  symbols 
belonging  to  the  DC  symbol  block  be  {b  { j) ,  where  the 
entry  at  (A,B)  in  M”  is  p.  It  is  evident  that  E  (a,b)  -"error" 
¥ae A  and  ¥b6B. 


Suppose  the  EEA  accessing  procedure  is  reversed:  ie-  the 
reduced  FSM  ft”  is  accessed  prior  to  accessing  E.  Proper  error 
detection  will  be  retained  if  the  entry  at  (A,E)  in  H”  is  p,  and 
the  detection  of  p  in  M"  is  treated  as  an  error  condition.  if 
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however  the  entry  at  {A,B)^0,  the  error  function  E(a,b)  must  be 
inspected  to  determine  the  legality  of  the  sta te/sy mho  1  pair. 
{If  E  (a,  b)  ^’’error’1  they  are  illegal;  if  E  (a,  b)  ="nc  error”  they 
are  legal.)  In  both  cases  (when  entry  (A,E)-p,  and  (A,B)Ap)  (a,b) 
will  be  correctly  error  checked.  Thus  the  error  function  values 
E{a,b)  will  never  be  accessed  when  the  entry  in  ft"  Is 
consequently  they  may  be  replaced  by  0.  The  simplification 
algorithm  is  thus: 

E(a,b)=£*  (¥aGA,  ¥bGB)  such  that  entry  (A  ,B)=p  in  H'b 
provided  the  reversed  error  accessing  mechanism  is  used. 

Because  the  error  function  is  an  incompletely  specified 
boolean  function,  it  may  be  reduced  in  size  by  applying 
incompletely  specified  boolean  minimization. 

Elimination  of  the  Error  Function 

In  the  construction  of  parsers  for  specialized  applications, 
there  are  occasions  when  error  detection  is  not  required  because 
the  input  text  is  known  to  be  correct  and  error-free.  Some 
typical  examples  are 

1)  A  parser  operating  in  the  second  or  third  pass  of  a 
multi-pass  compiler  that  parses  internal  compiler 
text ; 

2)  A  parser  driven  from  an  input  token  stream  prepared 
by  a  paragrapher  which  includes  a  parser. 
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In  both  examples  the  input  text  tc  the  parser  is  known  to  be 
error-free  since  it  has  already  been  checked  by  an  error¬ 
detecting  parser. 


Parsers  operating  on  correct  input  streams  may  benefit 
the  use  of  an  error-factored  matrix  structure  for  their  p 
tables  since  the  error  detection  function  may  be  deleted  enti 
in  these  instances. 
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Elimination  of  the  error  de 
implies  that  compatible  blocks  of 
Thus  the  definition  of  row  or 
instead  of  SEE  or  DC  in  the  for 
However,  as  pointed  out  in  Cha 
may  introduce  closure  restriction 
the  machine.  The  use  of  EC  elirai 
a  closed  solution,  hence  its  u 
compatibility  when  error  detect 
FSM. 

Factorization  of  Parser  Tables 

A  technique  of  space  reduct 
the  splitting  or  the  factoring  of 
machines.  These  component  machi 
of  each  other,  producing  a  set  of 
the  unfactored  set  for  a  grammar. 


tection  capability  from  a  parser 
a  FSd  cover  need  not  be  SEE. 
column  compatibility  may  be  used 
nation  of  the  machine  cover, 
pter  2,  the  use  of  compatibility 
s  on  the  set  of  cover  blocks  for 
nates  the  need  for  searching  for 
se  is  preferable  to  that  of 
ion  is  not  required  in  a  reduced 


ion  that  merits  consideration  is 
parser  tables  into  component 
nes  may  be  reduced  independently 
tables  that  may  be  smaller  than 
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The 


factorizations  invest 
Anderson  [1972],  and  a  variation 
[1969].  Anderson  factored  the 
tables  from  the  non-terminal  port 
tables  into  three  com ponent  machi 
the  look-ahead  machine,  and  the 
machine.  Because  the  parse  r 
shown  to  be  equivalent  to  the  non 
we  use  the  latter  in  our  variant 
of  reference,  call  the  terminal  - 
TNT  factorization,  with  the  termi 
and  the  non-terminal  machine  th 
modified  read-lock-restart  as 
terminal)  factorization,  where  th 
the  lock- ahead  is  the  L  machine, 
the  NT  machine. 

The  potential  space  savi 
considered  in  the  next  chapter. 

Performance  Criteria 

In  the  preceeding  sections  of 
approaches  to  the  minimization 
presented.  The  effectiveness  of 
measured,  and. compared  with  a  vie 
efficiencies. 


i gated  are  those  proposed  by 
of  that  proposed  by  DeRemer 
terminal  portion  of  the  machine 
ion;  DeRemer  factored  the  parse 
nes  -  the  terminal  read  machine, 
apply  production-parse  restart 
estart  component  machine  may  be 
-terminal  portion  of  the  table, 
of  the  factorization.  For  ease 
non-terminal  factorization  the 
nal  machine  being  the  T  machine, 
e  NT  machine.  Refer  to  the 
the  RLNT  (read,  look-ahead,  non- 
e  read  machine  is  the  R  machine, 
and  the  non-terminal  machine  is 
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The  experimental  verific 
straightforward.  Typical  grammar 
state  machines  produced.  The  siz 
compared  to  G LB (M 1 )  and  DCLB (M  * )  , 

a)  The  size  increase  of  red 
DC  instead  of  compatibil 

b)  Whether  Algorithm  #3  gen 

The  time  efficiency  of  Algori 

time  complexity.  However,  in  ord 
effectiveness,  the  generation  ti 
be  considered  and  compared.  Whil 
the  efficiency  of  their  implement 
principles  underlying  the  algorit 
extent . 
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The  space  effectiveness  of  the  various  transformations  ( BTC, 
CTR  reduction,  state  assignment,  chain  reduction,  etc.)  may  be 
demonstrated  by  comparing  transformed  table  sizes  to 
untransf ormed  table  sizes.  Typical  table  implementations  in 
bytes  may  also  be  used  as  a  valid  method  of  comparing 
transformation  effectiveness. 


Measuring  the  speed  ef f ect i veness  of  various  parser  table 
t ransf ormations  is  less  precise,  because  parser  speed 

measurements  are  extremely  implementation  and  computer  dependent. 
In  addition,  the  lack  of  a  suitable  "theoretic  lower  bound" 
against  which  parsing  speed  measurements  may  be  compared  further 
complicates  the  problem.  We  adopt  the  traditional  approach 
comparing  our  speed  results  against  those  produced  by  other 
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parsers  for  a  particular  programming  language,  where  all  parsers 
are  driven  from  a  common  input  stream,  and  all  tests  are 
performed  on  the  same  computer  under  the  same  circumstances. 

We  also  attempt  an  alternate  approach:  we  compare  the  speed 
of  matrix  lookup  with  that  of  a  list  lookup,  using  realistic 
parser  table  data.  A  relative  figure  of  merit  may  be  determined 
that  enables  an  effective  comparison  between  the  two  lockup 
methods  by  using  a  simple  statistical  result  and  some  timing 
figures  for  particular  lookup  implementations. 

The  above  results  are  presented  and  discussed  in  the  next 
chapter. 
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Chapter  4 


Results  and  Eiscussipn 


0  verview 


Since  the  objectives  of  this  investigation  have  been  to 
'produce  practical  error-detecting  reduced  matrix  parser  tables 
for  LR  (k)  grammars,  we  consider  in  this  section  to  what  extent 
our  goals  have  been  achieved.  The  case  study  method  is  adopted 
for  the  comparison  and  analysis  of  the  performance  criteria 
mentioned  at  the  end  of  the  last  chapter.  Rather  than  attempting 
to  form  a  "typical”  average  LR  (k)  grammar,  we  choose  a  simpler 
method  by  selecting  published  grammars  that  have  been  used  to 
demonstrate  some  facet  of  language  theory  or  parser  development. 
The  grammars  are  all  SLR{1).  Choosing  a  subset  of  the  more 
general  LR  (k)  grammars  for  illustrative  purposes  does  not  weaken 
the  results  presented  here,  since  the  reduction  algorithms 
presented  in  the  last  chapter  may  equally  well  be  applied  to 
LR{k),  LALR  {k )  ,  or  LALR(1)  grammar  parser  tables.  The  chain  rule 
elimination  algorithm,  however,  is  applicable  only  to  SLR(1)  or 
LR{1)  grammar  tables. 

The  principal  grammars  are: 

1)  An  arithmetic  expression  grammar  "AE"  [Pager  1970]; 

2)  Korenjak's  grammar  ”G"  [1969],  which  will  be  referred  to 
as  "KG"; 

3)  Pager's  grammar  "G”  [1972],  which  will  be  referred  to 
as  "PG"; 
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4)  The  reference  grammar  for  the  language  XPL 
[ McKeeman  et  al.  1970]; 

5)  The  grammar  for  the  language  SUSIE. 

The  first  three  grammars  were  chosen  primarily  for  their  s 


s  ize. 

The 

last  two 

were 

chosen  because 

they 

represen  ted 

class 

of 

"typical" 

progr 

arming  languages 

which 

the  algorithm 

the 

last 

chapter 

were 

designed  to 

pro 

cess.  The 

representations  for 

the 

first  three  gr 

ammars 

and  for  SUSIE 

contained  in  Appendix  I.  Table  4.1  contains  perti 
information  about  the  five  test  grammars.  Included  in  this  t 
is  the  related  information  for  ten  other  grammars.  T 
grammars  are  used  subsequently,  and  are  included  here 
completeness. 

The  five  principal  grammars  were  processed  by  the  SL 
grammar  analyzer  written  by  F.  L.  DeBemer.  The  tables  emi 
from  this  processor  have  been  modified  to  include  the  s 
transition  information  of  the  parsing  tables  in  a  mach 
readable  form.  The  transition  information  emitted  after 
grammar  analysis  was  accepted  as  input  by  an  XPL  program  w 
contained  efficient  encodings  of  Algorithms  #1  to  #3.  Under 
control  of  user-settable  toggles,  this  program  processed  the 
transition  information  by  any  of  the  three  algorithms,  fact 
and  reduced  the  error  function  matrix,  and  reduced  the  par 
matrix  by  rows,  columns,  or  both.  The  program  then  emitted 
reduced  parser  tables  and  the  error  function  in  the  form 
initialized  XPL  arrays,  suitable  for  insertion  into  a  general 
parser  framework. 
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While  the  i rap lementation  details  of  the  redaction  algorithms 
are  of  secondary  importance,  the  comparative  statistics  on  the 
reduction  processes,  and  the  sizes  of  the  reduced  machine  tables 
are  of  interest. 

Fow  Reduction  by  Algorithm  #1 

Using  both  the  definitions  of  compatibility  and  disjoint 
compatibility  to  generate  pairwise  incompatible  and  pairwise 
disjoint  incompatible  sets  of  states  respectively.  Algorithm  #1 
was  used  to  generate  reduced  machines  for  the  five  principal  test 
grammars.  The  Ginsburg  lower  bound  on  the  number  of  states  in 


the  minimal 

reduced 

machine  was 

also 

computed  using 

both 

ccmpa  tibilit y 

and 

DC 

def init ions . 

The  nu 

mber  of  potential  i 

mage 

and  SEB  errors 

was 

tabulated  using 

the  com 

patifcility  definit 

ion. 

as  was  the  actual  number  of  closure  and  5ED  errors  in  the 
selected  cover  machine.  These  results,  along  with  the  sizes  of 
the  reduced  row  machines,  are  shewn  in  Table  4.2.  The 
effectiveness  of  Algorithm  31  in  reducing  the  number  of  states  in 
the  table  representation  of  the  error-f act ored  parser  is  evident: 
the  reduced  cover  DC  machines  contain  from  17%  to  46 %  of  their 
original  number  of  states. 

Using  compatibility,  the  covers  for  the  first  four  parser 
machines  achieved  the  Ginsburg  lower  bound  on  the  number  of 
compatible  blocks  of  states  in  the  reduced  machine.  However,  in 
the  cases  of  PG  and  XPL,  the  sets  of  cover  blocks  selected  were 
not  closed.  Using  DC,  the  covers  selected  for  each  of  the  parser 
FSM*s  were  closed,  and  achieved  their  respective  BCLE  sizes. 
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G  rammar 

Pr-ods 

Trans  1 n  s 

State 

AE 

1 1 

75 

17 

KG 

8 

19 

8 

PG 

10 

28 

1  8 

X  PL 

109 

925 

101 

SUSIE 

71 

719 

102 

STAGG  . 

65 

504 

60 

MAL 

62 

754 

89 

FOTON 

63 

480 

66 

SIMPER 

80 

618 

85 

SNOGOL 

72 

701 

99 

TOY 

30 

181 

35 

EULER * 

92 

2577 

105 

DISML+ 

123 

1  197 

13  1 

SIMPAC+ 

128 

1323 

119 

ALGOL  60+  118 

1226 

134 

Vocab . 

Ter  m  *  1 

Non-term1 1 

Symbols 

Symbols 

Symbols 

13 

9 

4 

1  1 

6 

5 

17 

12 

5 

51 

42 

49 

66 

4  1 

25 

72 

36 

3  6 

67 

33 

3  4 

62 

36 

26 

87 

52 

3  5 

67 

44 

23 

38 

26 

12 

97 

6  6 

3  1 

107 

46 

6  1 

117 

50 

67 

10  1 

49 

52 

Table  4.1  -  Grammar  Information 


*  The  structure  cf  the  grammar  has  been  altered  so  that 

fprods  +  #parse  states  <  256  due  to  a  design  restriction 
of  the  SLR  ( 1 )  analyzer. 


# 

Closure/SED 

#  DC 

Cover  Errors 

Cover 

G  rammar 

#Sta  tes 

GLB 

Ploc  ks  Pot 1 1 

Actua  1 

DCLB 

Blocks 

AE 

17 

8 

8 

>  20 

0 

8 

8 

KG 

8 

3 

3 

2 

0 

3 

3 

PG 

18 

3 

3 

4 

3 

3 

3 

XPL 

101 

43 

43 

>240 

8 

46 

46 

Table 

4.2  - 

Alg.  #1 

State 

Reduction 

Results 

Decomp. 

Search 

Total 

Total  #  DC 

Grammar 

Time 

Time 

Time 

Blocks  Gent' 

1 

AE 

1.90 

0.65 

2.55 

76 

KG 

0.81 

0.25 

1.06 

12 

PG 

25.59 

1 . 45 

27.04 

486 

XPL 

501.66 

1787.74 

2289.40 

125640 

SUSIE 

9 

•9 

9 

>  520032 

Table  4.3  - 


Alg.  #1  Time  -  Workspace  Requirements 
(Elapsed  time  in  seconds) 
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The  Ginsburg  lower  bound  on  the  number  of  state  blocks  in  a 
minimal  reduced  machine  is  an  estimate.  The  ability  to  calculate 
this  bound  does  not  guarantees  that  an  acceptable  FS M  can  be 
found  that  achieves  this  bound.  If  a  solution  is  produced  for  a 
FSM  that  reaches  this  lower  bound,  it  represents  one  of  a 
possible  multiplicity  of  acceptable  solutions  for  the  reduction. 
If,  however,  a  solution  is  produced  that  exceeds  the  bound,  no 
conclusion  can  be  reached  concerning  the  attainability  of  a 
ninimal  solution.  In  a  similar  fashion,  the  presence  of 
potential  image  conditions  among  the  states  cf  an  unreduced  FSM, 
or  the  presence  of  actual  closure  conditions  among  compatible 
state  blocks  of  a  reduced  cover,  does  not  imply  that  the  GLB  is 
unattainable  for  the  machine.  Thus  the  presence  of  eight  closure 
errors  in  the  cover  set  of  state  blocks  for  the  XPL  machine  does 
not  mean  that  the  GLB  of  43  compatible  state  blocks  is 
unachievable  -  just  that  it  must  be  searched  for  among  the  set  of 
acceptable  solutions.  This  search  is  obviated  by  the  use  of  DC, 
at  the  expense  of  adding  three  state  blocks  to  the  XPL  closed 
cover  machine.  It  is  known  that  a  43  block  SEE  closed  cover 
cannot  exist,  since  the  SEEIB  is  the  DCLB  for  the  EeBenter  SLR 
FSM.  Thus  the  46  block  solution  is  theoretically  minimal. 

Problems  with  Algorithm  #1 

While  Algorithm  #1  produced  acceptable  solutions  for  the 
reduced  covers  of  the  test  grammars,  the  computation  time 
reguired  was  excessively  high.  The  extensive  enumeration  and 
workspace  needed  for  the  algorithm  is  reflected  in  Table  4.3, 
where  the  elapsed  decomposition  time,  the  elapsed  search  time. 
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ime,  and  the  number  of  DC  blocks  generated  and 
u la ted.  The  timing  statistics  are  not  known  for 
analysis,  because  the  computer  execution  of  the 
was  terminated  after  more  than  half  a  million  DC 
enerated,  after  approximately  40  minutes  of 
timing  figures  of  Table  4.3  were  obtained  from  a 
on  an  IEM  360  model  44,  under  the  OS/360  MFT-II 
The  program  was  the  only  active  task  executing 
en  the  tests  were  performed;  all  other  tasks 
The  timing  results  were  repeatable  to  within  one 


For  the  three  small  grammars,  the  execution  time 
the  operation  of  Algorithm  #1,  and  the  number  of 
developed  in  the  solution  process  are  not  unreason 
for  X PL  and  SUSIE  are  a  different  story:  they  are  bot 
The  absolute  timing  figures  for  the  reduction  o 
grammars  and  the  workspace  requirement  poin 
impractica lity  of  Algorithm  #1  for  the  reductio 
programming  language  parser  tables. 
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Sow  Reduction  by  Algorithm  #2 


Using  the  definition  of  DC,  Algorithm  #2  was  used  to  generate 
reduced  state  closed  cover  machines  for  the  five  test  grammars. 
The  DC  lower  bound  for  each  was  also  produced.  The  cover  size 
results,  and  the  generation  time-workspace  requirements  for  the 
execution  of  Algorithm  #2  are  shown  in  Table  4.4. 
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Grammar 

DCLE 

#  DC 
Cover 
Blocks 

Net  Elapsed 

Time  (sec. ) 

Total  #  DC 
Blocks  Gen 

AE 

8 

8 

0.37 

27 

KG 

3 

3 

0.07 

9 

PG 

3 

3 

0.30 

286 

XPL 

46 

46 

3.74 

1047 

SUSIE 

43 

44 

3.15 

756 

Table  4.4 

-  Alg. 

#2  State 

Reduction  Results 

and 

Time  -  Workspace  Requirements 


Grammar 

Alg 

.  #2/#3 

Deco  mp 1 n 

Steps  | 

State 

#DC  Cover 

Alg 

if  3 

Exec  1 n 

Predicted 

Actual 

I 

DCLE 

Blocks 

Time 

{ 

sec. ) 

AE 

13 

8 

1 

1 

8 

8 

0. 

18 

KG 

4 

3 

I 

3 

3 

0. 

10 

PG 

11 

3 

1 

3 

3 

0. 

1  1 

XPL 

90 

46 

1 

46 

46 

2. 

69 

SUSIE 

83 

44 

1 

43 

44 

2. 

39 

STAGG 

4  9 

29 

I 

29 

29 

— 

— 

HAL 

89 

37 

\ 

37 

37 

— 

— 

FOTON 

51 

28 

28 

28 

— 

— 

SIMPER 

67 

41 

\ 

4  1 

4  1 

— 

— 

SNOGOL 

83 

44 

1 

44 

44 

— 

— 

TOY 

24 

18 

1 

18 

18 

— 

— 

EULER* 

94 

53 

1 

53  . 

53 

2. 

92 

DISML* 

1  18 

57 

1 

55 

57 

4. 

3  1 

SIMPAC* 

107 

59 

1 

59 

59 

3. 

68 

ALGOL  60* 

112 

55 

1 

J 

50 

55 

4. 

01 

Table  4 

.5 

-  Predicted 

vs 

Actual  | 

Table 

4.6  - 

Al  g 

.  #3 

Number  of 

Deccmp 1 n  ] 

State  Red 

’  n 

Steps 

for 

Alg 

#  2/ #  3  | 

1 

Re  s 

ult  s 

Grammar 

Symbol 

AE 

8 

KG 

3 

PG 

11 

XPL 

30 

SUSIE 

25 

#  DC  Symtcl 
DCLE  Cover  Elocks 


8 
3 
1  1 

30 

25 


Table  4.7  -  Alg .  #3  Symbol 

Red’n  Results 
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R  e  duced 
grammars  and 
rows;  that 
Algorithm  #2 


cover  machines  were  processed  for  the  three  small 
X PL  that  contained  a  theoretically  minimal  number  of 
for  SUSIE  exceeded  the  DCLB  by  one  block.  Thus 
produced  acceptably  reduced  error-factored  machines. 


The  elapsed  computation  t 
for  the  reduction  computations 
algorithm  in  an  absolute  sense, 
corresponding  figures  for  Algcri 
show  a  great  improvement.  While 
the  results  of  the  small  grammar 
larger  grammars  are  pronounced, 
considerable  performance  improve 
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Practical  Complexity  of  Algorithm 


As  previously  shewn.  Algorithm  #2  has  a  worst  case 
complexity  that  is  exponential  in  the  number  of  states  to  be 
decomposed,  as  does  the  less  efficient  Algorithm  #1.  From  the 
practical  evidence  of  the  test  grammars,  the  former  is  much  more 
efficient  than  the  latter.  Part  of  this  increased  efficiency  is 
due  to  the  overlapping  of  the  initial  set  of  pairwise  compatibles 
with  the  states  of  DC  blocks  selected  in  a  prior  decomposition 
step.  Another  source  of  improved  performance  is  the  early 
termination  of  the  major  decomposition  loop  due  to  step  4)  of  the 
algorithm.  This  test  decreases  the  total  number  of  decomposition 
steps  significantly,  as  is  shown  in  Table  4.5,  where  the 
predicted  number  of  decomposition  steps  is  tabulated  with'  the 
actual  number  of  decomposition  steps  for  the  test  grammars.  For 
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the  larger  grammars,  the  actual  number  of  major  decomposition 
steps  performed  is  approximately  half  the  value  predicted.  Since 
the  predicted  value  is  the  number  of  decomposition  steps  that 
will  be  followed  by  Algorithm  #1,  the  decrease  in  the  actual 
number  of  steps  for  Algorithm  #2  is  an  indication  of  its  improved 
performance:  the  magnitude  of  the  exponent  in  the  complexity 
estimate  for  Algorithm  #2  is  half  that  for  Algorithm  #1. 

Bow  Reduction  by  Algorithm  S3 


Using  the  state  DC  definition.  Algorithm  #3  was  used  to 
generate  reduced  covers  of  the  fifteen  grammars  detailed  in  Table 
4.1.  The  DC  lower  bound  for  each  was  also  produced.  The  cover 
size  results  and  corresponding  DC1E  values 
4.6.  Appendix  VII  contains  an  illustrate 
generation  for  KG  by  Algorithm  #3. 
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sacri f iced  the  ability  of  Algorithm 
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#3  to  achieve  acceptably 


The  elapsed  Algorithm  #3 
principal  test  grammars  is  al 
differences  between  the  results 
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#2. 

Algorithm  #3  has  been  sho 
procedure  for  rapid  generation 
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Pager  [1971]  has  also  used  Korenjak*s  grammar  to  demonstrate 


the  reduction  of 

Knu th  *  s 

parser  tables. 

and  has 

provided  a 

reduced  solution 

of  six 

state  compatibl 

e  blocks. 

Our  reduced 

solution  was  half 

the  size 

of  Pager’s,  and 

preserved 

the  error 

detection  capabil 

ity  of  the 

machine  whereas 

Pager's  d 

id  net. 

Pager  [1970]  used  the  AE  grammar  to  demonstrate  the 
possibility  of  minimizing  parser  tables.  His  unreduced  machine 
contained  28  states,  and  his  minimal  reduced  one  contained  7 
state  compatibles.  We  correct  a  minor  error  in  Pager’s  formation 
of  the  reduced  machine  by  stating  that  the  cover  should  have 
contained  8  state  compatibles.  Pager’s  AE  parser  tables  and  the 
tables  produced  by  the  SLE(1)  analyzer  both  contain  17  (non- 
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From  these  results.  Algorithm  # 
reduced  machines  that  are  comparable 
previously  published  results,  as  well 
development  of  these  results. 

Column  deduction  by  Alg.  #3 

The  five  principal  test  grammar 
reduced  machines  composed  of  DC  symbol 
DCLB  was  also  determined  for  the  five 
Table  4.7  show  that  the  number  of 
generated  equalled  the  symbol  colum 
grammars. 


Practical  RTC  and  CTP  Reduction 


Although  RTC  and  CTR  reductions 
optimally  reduced  solution  machines  M” 
reduced  machines.  The  reduction  resu 
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as  being  applicable  in  the 


s  were  processed  to  find 
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grammars.  The  results  in 
DC  column  cover  blocks 
n  ECIE  for  the  five  test 
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nd  summarized  in  Tables 
the  original  table  space 
achine  for  each  grammar 
space  are  possible  in  the 


In  every  case  of  RTC  reduction,  the  column  reduction 
performed  on  the  reduced  row  matrix  achieved  the  symbol  DCLB  for 
the  reduced  row  matrix.  Likewise,  for  CTR  reduction,  the  row 
reduction  on  the  reduced  column  matrix  produced  a  solution  having 
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the  same  number  of  row  DC  blocks  as  the  state  OCLB.  These 
results  show  that  RTC  and  CTR  reduction  produce  machines  that  are 
as  reduced  as  possible  for  sequential  reduction  processes  of  this 
type. 


w 

ith 

produced 
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table 

s,  s 

apply 

sta 
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nali 

Thus, 

if 

rows 

ha  vi 

DC  bl 

ock . 

B 

ecau 

or  in 

mer 

ident. 

ical 

trans 

itio 

the  test 


the  exception  of  the  SUSIE  grammar,  RTC  reduction 
smaller  parser  matrices  than  did  CTS  reduction.  This 
s  anticipated  because  of  the  structure  of  SLR(1)  parsing 
ince  as  previously  mentioned  all  transitions  to  rou¬ 
tes  occur  in  one  column  of  the  table*  This  implies  more 
ty  of  state  numbers  in  any  column  than  in  any  row. 

row  reduction  were  performed  prior  to  column  reduction, 
ng  identical  read  transitions  might  be  merged  into  one 

se  of  the  DC  criterion,  the  only  overlap  in  merged  rows 
ged  columns  of  a  table  occurs  when  merged  entries  are 
.  This  results  in  a  potential  decrease  in  the  number  of 
n  entries  in  the  reduced  table.  Such  is  the  case  for 
grammars  (see  table  4.10). 


Organization  of  the  SL5(1) _ Matrix  Parser  Tables 


The  table  organization  of  a  matrix  parser  employing  the 
unreduced  flow  table  is  shown  in  Figure  4.1.  The  machine  M  is 
err or— factored  into  (M*,E) •  These  two  tables  are  then  stored 
separately.  If  ( M  1 ,  E)  is  reduced  to  f  the  former  by  Alg. 
#3  using  RTC  or  CTR  reduction,  and  the  latter  by  boolean  function 
techniques  (eg-  merging  identical  rows  and  columns),  indirection 
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vectors  (the  "Index"  napping  of  Theorem  2)  must  be  maintained  to 
provide  mappings  from  the  unreduced  row  or  column  numbers  to 
those  in  the  reduced  arrays.  The  structure  of  a  reduced  machine 
matrix  N"  with  its  indirect  address  pointer  vectors,  and  a 
reduced  error  attribute  matrix  E1  with  its  pointers,  is  shown  in 
Figure  4.2.  All  accesses  to  table  data  are  dene  using  one  level 
of  indirection  via  the  table  pointers. 

Each  specified  entry  in  array  M 1  (and  also  in  array  MM) 
contains  two  separate  items: 

1)  An  absolute  state  transition  entry  referencing  the 
state  numbers  in  machine  Mf; 

2)  A  flag  entry  describing  the  action  to  be  performed 
with  the  current  input  token:  a  read  action  or  a 
look-ahead  action. 

The  states  of  M  are  numbered  sequentially.  If  there  are  Mp" 
productions  in  the  grammar,  and  "q11  states,  all  LR  (0)  reduce 
states  in  the  machine  are  numbered  from  1  to  p,  and  all  ncn-apply 
states  are  numbered  from  p+1  to  n,  where  n=p+g.  The  first  p 
apply  production  states  are  not  encoded  into  the  parsing  table. 
Thus  the  row  indirect  address  vectors  for  M"  and  E*  need  only 
contain  "g"  entries  each. 
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d  jaojt  i  on  CT  R  Reduction 


G  rammar 

rows 

* 

cols 

=  entries 

rows 

* 

eels  = 

entries 

AE 

8 

❖ 

10 

-  80 

15 

* 

8  = 

120 

KG 

3 

* 

8 

=  24 

8 

* 

3  = 

24 

PG 

3 

* 

17 

=  51 

18 

n' 

1  1  = 

193 

XPL 

46 

* 

40 

=  1840 

71 

❖ 

30  = 

2130 

SUSIE 

44 

* 

3  4 

=  1496 

58 

* 

25  = 

1450 

Table 

4.8  - 

RfC 

and 

CTR  Reduction 

by 

Alg 

.  #3 

Grammar 


Min  deduced  Size  / 
Unreduced  Size 


AE 

KG 

PG 

XPL 

SUSIE 


3  6.2  % 
27.3  % 
16.7  % 
20.5  % 
2  1.6  % 


Table  4.9  -  Reduction  Summary 

(Min.  Reduced  Size  = 
smallest  of  ETC  or  CTR 
reduced  table  sizes) 


G  ramrnar 


Unreduced  Table 
#Tra  ns^_n  s  Info. _ Density 


Reduced  Table 
^Trans'  ns  Info.  Density 


AE 

75 

3  4.0  % 

70 

87.5 

% 

KG 

19 

21.6  % 

18 

75.0 

0/ 

/O 

PG 

28 

0.9  % 

28 

54. 9 

& 

/3 

XPL 

925 

10.1  % 

896 

48.8 

% 

SUSIE 

719 

10.6  % 

671 

46.  6 

& 

Table 

4.10  - 

Transition  Table 

I nforma t ion 

Densities 
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Fiqure  4.1  -  Error  Factored  Structure 
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Fiqure  4.2  -  Reduced  Matrix  Parser  Structure 
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Parser  Table  Size  Results 


The  preceding  information  about  the  indirect  address  vector 
dimensions,  the  reduced  matrix  sizes  for  M"  (Table  4.8),  and  the 
reduced  matrix  sizes  for  f1  (Table  4*11)  may  be  used  to  calculate 
the  sizes  of  a  parser  implementation  for  the  test  grammars.  The 
host  computer  for  the  implementation  was  the  I  EM  360/370  series, 
primarily  because  of  its  availability  at  the  University  of 
Toronto,  and  because  the  widespread  use  cf  parser  generators  for 
these  machines  invites  comparisons  with  previously  published 
results.  The  implementation  considerations  are  presented  in 
Appendix  II. 


The 

pars 

er  table 

i mple mentation 

space 

requirements  for 

the 

five  test 

gra 

mmars  are 

shown  in  Table 

4.  12. 

These  figures  do 

not 

Include 

the 

grammar 

vocabulary,  wh 

5ch  is 

reguired  only  du 

ring 

error  diagnosis  of  an  invalid  parse  sequence. 

The  space  required  for  the  reduced  matrix  parsing  t 
of  reasonable  magnitude.  However,  the  table  sizes 
compared  to  those  produced  by  accepted  parser  generators 
/360:  the  MSP  parser  generator  used  In  the  XPL  compiler 
system  [ McKeeman  et  al.  1970],  and  the  LALR  parser  g 
[ Lalonde  1971]  which  was  designed  to  replace  it.  F 

results  of  Lalonde,  the  MSP  and  LALR  parser  tables  for  XP 
3274  bytes  and  1250  bytes  respectively,  compared  vith  301 
for  the  reduced  matrix  tables. 


ables  is 
must  be 
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writing 
ener a tor 
rom  the 
L  occupy 
2  bytes 


-78- 


Urre  duced 

Table 

Reduced  Table 

Reduced  Size 

Grammar 

rows 

eels 

rows 

cols 

Ib^tes} 

AE 

17 

9 

6 

7 

6 

KG 

8 

6 

6 

7 

6 

PG 

18 

12 

7 

8 

7 

XPL 

10  1 

42 

47 

29 

188 

SUSIE 

102 

41 

47 

33 

235 

Table 

4.11  -  Error  Table 

Sizes 

(terminal 

symbols  only) 

Grammar 


Parser  Table  Size 
ibytes] 


AE 

KG 

PG 

XPI 

SUSIE 


197 
90 
170 
30  12 
2472 


Table  4.12 


Reduced  Matrix  Parser 
Table  Sizes 


From  the  size  result  comparison  for  the  XPL  grammar,  the 
matrix  tables  are  much  larger  than  the  LALR  produced  tables. 
Part  of  the  reason  for  this  lies  in  the  space  compaction  methods 
used  in  the  table  generator  implementations.  LALR  represents  the 
grammar  parser  table  as  a  list  structure,  and  uses  list 
optimization  techniques  tc  minimize  the  number  of  unique  list 
entries.  Such  techniques  as  the  overlapping  of  list  subsets, 
used  in  the  LAIR  generator,  cannot  be  used  in  the  matrix  table 
generator.  In  addition,  the  use  of  an  "in  any  other  case" 
transition  to  represent  the  most  popular  destination  state  in  the 
list  cannot  be  used  if  the  parsing  table  is  stored  as  a  matrix. 


The  list  representation 
pointer  into  the  grammar  voca 
the  compacted  list,  filling  o 
transition  data.  Provided 
overlapped  or  replaced  by 
entry,  the  50%  space  overhead 
outweighed  by  the  reduction  i 
is  the  case  in  the  LALR  table 
contain  511  transitions,  regu 
and  pointers.  The  reduced  ma 
entries  in  the  matrix,  for  an 
less  than  that  for  the  list 
reduced  matrix  table  represen 
of  the  matrix  table  pointer  s 
the  read/look-ahead  flag  m 
the  matrix  representation  for 
is  not  as  compact  as  the  LALR 


used  in  the  LALR  tables  requires  one 
bulary  for  each  transition  entry  in 
nly  half  the  list  storage  space  with 
enough  transition  entries  may  be 
the  "in  any  ether  case"  transition 
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of  table  entrie 

s. 

Such 

s  for  XPL: 
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iring  1022  bytes  of  storage 

for 

data 

trix  for  XPL, 

however,  conta 
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896 

information 

data  density  of 

48. 

8%  - 

representation.  When  the  less- 
taticn  is  added  to  the  high  overhead 
pace,  the  error  matrix  space,  and 
atrix  space,  it  becomes  apparent  why 
the  parsing  tables,  for  XPL  anyway, 
list  representation. 
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Figure  4.3  -  State  Assigned  Reduced  Matrix  Parser  Structure 

(Error  det*n  matrix  shown  in  Figure  4*2) 
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RTC 

Reduction 

CTR 

Reduction 

Grammar 

rows 

*  cols  =  entries 

rows 

*  cols  -  entries 

AE 

6 

7 

42 

8 

5  40 

KG 

3 

6 

18 

7 

3  21 

PG 

3 

3 

9 

4 

2  8 

XPL 

34 

25 

850 

41 

15  615 

SOSIE 

28 

22 

616 

34 

14  476 

Table 

4.13  - 

Reassigned 

Reduced 

Parser 

Matrix  Sizes 

Grammar  af 

b° 

AE 

36.2  % 

18.1  %  2.0 

KG 

27.3  % 

20.5  %  1.3 

PG 

16.7  % 

2.6  %  6.4 

XPL 

20.5  % 

6.7  %  3.1 

SCSI  E 

21.6  % 

7.1%  3.1 

Table  4.14  - 

State  Assignment 

Improvement  Ratio 

+a)  unassigned 

table  size  red’n 

°b)  state  assigned  size  red*n 

Grammar 

#  Transitions 

Info.  Density 

AE 

31 

77.5  % 

KG 

15 

83,3  % 

PG 

5 

62.5  % 

XPL 

404 

65.7  % 

SOSIE 

325 

68.2  % 

Table  4.15  - 

Reassigned  Reduced  Matrix  Info.  Density 

Grammar 

#  Apply  List 
Entries 

Parser  Table  Size 
(bytes) 

AE 

13 

184 

KG 

8 

108 

PG 

10 

163 

XPL 

167 

1836 

SUSIE 

177 

1611 

Table  4.16  -  State  Assigned  Reduced  Parser  Table  Sizes 
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If  the  matrix  parser  is  going  to  compete  with  a  LALE 
generated  list  parser  insofar  as  parser  table  space  is  concerned, 
some  optimizations  will  have  to  be  performed  that  reduce  the 
storage  requirements  for  the  matrix  parser  data  structure.  One 
such  technique  is  the  state  assignment  process,  described 
previously. 

State  Assignment  Results 

The  relationship  among  the  components  of  the  state  assigned 
parser  matrix  structure  after  reduction  is  shewn  in  Figure  4.3. 
The  structure  does  not  include  the  error  matrix  E,  since  it  is 
not  affected  by  the  state  assignment  procedure. 

The  sizes  obtained  for  the  reassigned  parser  matrices  after 
ETC  and  CTR  reduction  are  shown  in  Table  4.13.  The  reduced  sizes 
obtained  for  all  machines  are  considerably  smaller  than  those 
obtained  without  the  state  assignment  procedure.  Table  4,14 
contains  a  summary  of  these  results  for  comparison  purposes. 
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of  non-apply  transition's  took  better  advantage  of  the  underlying 
table  structure  than  did  the  row  assignment  cf  apply  transitions. 

Along  with  the  improvement  in  matrix  size  reduction,  there 
was  a  corresponding  general  increase  in  tie  information  density 
of  specified  transitions  in  the  matrix,  and  a  decrease  in  the 
number  of  specified  transitions  when  the  assignment  procedure  was 
employed  prior  to  reduction.  For  example.  Table  4.15  shows  that 
for  XPL  the  number  of  specified  transitions  in  the  smallest 
reassigned  reduced  matrix  was  404,  yielding  an  information 
density  of  65.7%,  compared  with  896  specified  transitions  for  a 
density  of  48.8%  in  the  smallest  unassigned  reduced  matrix. 

The  effectiveness  of  the  assignment  algorithm  in  increasing 
commonality  among  rows  and  columns  cf  the  parser  matrix  may  be 
illustrated  by  some  of  the  column  DC  blocks  created  in  the 
reduced  cover.  For  XPL,  seme  of  the  terminal  symbols  overlapped 
after  assignment  and  reduction  were: 

1)  *  - 

2)  *  /  MOD 

3)  IF  DO  RETURN  CALL  GO  DECLARE  WHILE  CASE 
INITIAL  (  END 


but  FIXED, 

CHARACTER, 

LABEL  were  not 

overl eppe  d 

because  the 

a 

pply 

production 

t ransi tio 

ns  were  not 

reassigned 

to  permit 

t 

heir 

merging. 

In  SUSIE, 

seme  of  the 

terminals 

overlapped 

a 

f  ter 

assignment  and  reduction  were: 

1)  TYPE  PROCEDURE 

2)  POINTER  ARRAY 

3)  EXIT  REPEAT  RETURN  CASE 
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4) 


H  HEN 


UNLESS 


but  +  ,  -  were  not  overlapped,  nor  were  ■*,  /,  KOD  merged. 

In  the  above  cases  where  merging  was  net  performed  on 
terminal  symbol  columns,  the  terminals  appreared  alone  of  the  PHS 
of  a  grammar  production,  rather  than  as  one  element  of  a  length- 
decreasing  production.  For  proper  processing  by  the  assignment 
algorithm,  these  terminals  should  have  been  substituted  into 
length-decreasing  productions,  thus  eliminating  productions  such 
as 

<adding_operator >  ::=  +  ] 

as  found  in  SUSIE.  The  effect  of  this  substitution,  however, 
would  be  to  increase  column  mergeability  at  the  expense  of  adding 
more  states  to  the  parser  FSM.  The  tradeoffs  of  such 
substitutions  must  be  investigated  more  fully.  Ue  leave  this 
topic  for  future  consideration.  It  Is  noteworthy  that  terminal 


symbols  which  were 

sy n tac  tica 

lly  interchangeable 

In  a 

language 

(such  as  ♦  and 

-  in  XPL) 

were  merged  into  a 

common 

DC  column 

block  because  cf 

the  assi 

gnment  procedure. 

Wit  ho 

ut  this 

assignment,  such  merging  was  not  performed. 

If  a  measure  of  an  algorithm’s  effectiveness  can  be  formed  by 
comparing  its  results  against  intuitive  results,  then  consider 
those  of  Algorithm  #3  performing  column  reduction  on  XPL.  We 
choose  column  rather  than  row  reduction  since  the  similarity 
between  two  columns  is  simpler  to  grasp  because  of  the  way 
symbols  are  used  in  a  language.  By  intuition,  such  symbols  as  + 
and  -  should  be  overlapped;  by  Alg.  #3  after  assignment  these 
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with  these  were 


symbols  were  overlapped, 
the  symbols 


however 


along 


merged 


PROCEDURE  LABEL  <statement>  <terra>. 

The  reason  for  their  merger  is  not  intuitive,  since  the  reduction 
algorithm  typically  merges  symbols  that  either  are  equivalent,  or 
that  are  totally  dissimilar. 


Perhaps 

the 

most 

unusual 

block 

formed 

was  the  second  DC 

column  block 

in 

the  CTR 

reduction 

of 

the 

as 

signed  XPL  parser 

machine.  Th 

is 

block 

contained 

31 

DC 

sym 

bols.  The  terminal 

symbols  were  given  previously  in  example  3)  above  for  XPL .  Along 
with  these  eleven  terminals  were  merged  the  following  twenty  non¬ 
terminal  symbols: 

<program>  <group>  <procedure  definition>  <return  statements 
<call  statement>  <go  to  statement>  declaration  statement> 

<if  clause>  <true  parts  <group  head>  <step  definition> 

<while  clause>  <case  selector>  <procedure  head> 

<procedure  name>  <go  to  >  <initial  head>  <left  part> 
<relation>  <subscript  head> 

Obviously  the  production  of  a  block  as  complex  as  this  was  far 
from  intuitive! 

Reassigned  Reduced  Parser  Table  Size  Results 

The  implementation  sizes  in  bytes  of  the  reassigned  reduced 
parser  tables  has  been  tabulated  in  Table  4.16  for  the  five  test 
grammars,  along  with  the  number  of  list  entries  in  the  production 
number  data  list  after  the  overlapping  of  identical  sublists. 
The  implementation  considerations  are  contained  in  Appendix  III. 
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size  as  much  as  possible  than  it  is  for  the  small  ones. 
Comparing  the  assigned  reduced  parser  table  size  o£  1836  bytes 
for  XPL  with  that  produced  by  the  LALR  parser  generator  (1250 
bytes)  indicates  that  the  assigned  reduced  matrix  tables  are  now 
more  competitive  in  size  with  the  list  structured  tables. 


Error  Function  Reduction 


In  order  to  reduce  the  space  required  by  the  matrix  parser 
tables  further,  more  extensive  reductions  may  be  performed  on  the 
error  function  E.  Ey  introducing  "don1!  cares1’  into  the  error 
function,  and  then  applying  incompletely  specified  boolean 
function  minimization  to  the  resulting  table,  greater  compaction 
of  the  error  function  may  be  done.  The  techniques  for 
introducing  these  $  entries  were  discussed  in  the  last  chapter. 
An  incompletely  specified  minimization  process  applied  to  the 


error  function 

matrix 

might 

reduce  the  size  of 

the 

matrix 

considerably  below  that 

of 

the  completely  specifi 

ed 

matrix. 

However,  there 

are  economic 

disadvantages  in  using 

the 

’’don’t 

cares”  and  the  incompletely  specified  reduction: 
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function  matrix  occupies  188  bytes  of  storage  space.  The  most 
that  could  be  gained  by  using  an  incompletely  specified  algorithm 
would  be  to  eliminate  these  188  bytes  completely,  or  more 
realistically,  to  cut  the  reguired  storage  space  by  perhaps  80%. 
In  the  context  of  reduced  XPL  parsing  table  sizes  of  3012  and 
1836  bytes,  this  savings  in  table  space  amounts  to  at  most  6%  and 
10%  of  the  total  table  space. 


If  maximal  reductions  are  to  be  performed  whenever  possible 
in  order  to  minimize  table  space,  the  net  savings  in  table  space 
using  "don’t  cares”  and  incompletely  specified  boolean 
minimization  would  be  worth  the  effort.  However,  in  the  context 
of  this  investigation,  its  implementation  is  not  essential.  We 
thus  leave  the  practical  implementation  problems  for  future 
consideration,  as  well  as  the  actual  space  reductions  that  may  be 
expected  from  use  cf  this  technique  of  optimization. 
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Error  Function  Elimination 


As  discussed 
input  streams  do 
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the  error  matrix 
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matrix . 


Eliminating  error  detection  has  practical  speed  imp! 
for  the  parser:  the  overhead  required  in  the  error  chec 
eliminated,  thus  allowing  the  parser  to  operate  more 
This  speedup  will  be  discussed  subsequently. 


Chain  Production  Elimination  Space  Results 


To  illustrate  the  space  saving  made  possible  by  the 
chain  production  elimination  algorithm,  ten  sema 
productions  were  eliminated  from  the  XFL  grammar  parse 
As  a  result,  three  read  states  were  not  accessed  in  the  u 
parser  matrix  M*.  The  chain-production  eliminated  tab 
then  reduced,  using  both  unassigned  and  state  assigned 
The  unassigned  reduced  matrix  M"  dropped  in  size  from 
1840  entries  to  42x41  or  1763  entries,  and  the  state 
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reduced  matrix  frcm  41x15  or  615  entries  to  43x13  or  559  entries. 


The  saving  in  table  space  is  not  significant  enough  to  make  chain 
production  elimination  worthwhile  if  it  were  only  a  space 
optimization  technique.  However,  since  its  chief  advantage  is  in 
parser  speedup,  the  space  improvement  in  the  reduced  tables  is  a 
valuable  side-ef f ect. . 

Factorization  Space  Results 

The  TNT  and  the  RLNT  factorizations  fcr  the  five  principal 
test  grammars  were  processed,  using  the  state  assignment.  The 
size  results  for  the  TNT  factorization  are  given  in  Table  4.17, 
and  those  for  the  RLNT  factorization  in  Table  4.18.  The  smallest 
of  the  tables  produced  by  ETC  or  CTR  reduction  has  been  used  in 
each  case.  The  calculation  of  the  space  requirements  for  the 
factorizations  is  presented  in  Appendix  IV. 


-90- 


T  Machine 


Grammar  rows  cols 


Total 

rows  cols  #  Entries 


AE  3  3 

KG  3  5 

PG  4  1 

XPL  21  11 

SUSIE  17  14 


6  4  3  3 

2  3  21 

1  2  6 

24  11  495 

20  7  378 


Table  4.17  -  TNT  Factorized  Reduced  Assigned  Matrix  Sizes 


R  Machine 

LA  Machine 

NT  Machine 

Total 

Grammar 

rows 

cols 

rows 

cols 

rows 

cols 

#  Entrie 

AE 

2 

3 

1 

2 

6 

4 

32 

KG 

2 

4 

1 

2 

2 

3 

16 

PG 

4 

1 

0 

0 

1 

2 

6 

XPL 

17 

6 

7 

1 1 

24 

1  1 

443 

SUSIE 

11 

7 

7 

13 

20 

7 

308 

Table  4,18  -  RLNT  Factorized  Reduced  Assigned  Matrix  Sizes 


Grammar 

Un  factored 

TNT 

RLNT 

AE 

184 

209 

232 

KG 

108 

127 

132 

PG 

163 

197 

209 

XPL 

1836 

1887 

1966 

SUSIE 

1611 

1678 

17C8 

Table  4. 19 

-  Factorized  As 
Parser  Table 

signed 

Sizes 

Reduced 
(in  bytes) 
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The  evaluation  of  the  implementation  si 
RLNT  factored  machines  is  shown  in  ' Table  h 
principal  test  grammars.  The  unfactored  par 
included  for  comparison  purposes.  For  all  f 
RLNT  factored  parser  tables  occupied  the 
space;  the  TNT  factored  table  requirements  wer 
the  RLNT  and  the  unfactored  parser  tables. 
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To  reduce  the  table  sizes  of  the  TNT  or  RLNT  factored 
machines,  certain  optimizations  may  be  performed  that  are  not 
applicable  to  the  unfactored  machine  tables.  For  example,  the 
states  of  the  unfactored  machine  matrix  may  be  sorted  and  renamed 
prior  to  factorization  such  that  all  states  with  non-terminal 
transitions  are  numbered  with  sequentially  increasing  state 
numbers.  The  same  may  be  done  for  the  terminal  look-ahead 
columns,  etc.  These  optimizations  help  to  reduce  the  dimension 
of  the  pointer  vectors  by  a  marginal  amount,  and  might  result  in 
the  factored  implementations  being  smaller  in  size  than  the 
unfactcred  tables. 
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There  is  little  to  choose  a 
the  three  machine  structures;  the 
HINT  factored  machines.  If 
considered,  then  the  unfactored  m 
choice.  If  the  ease  of  gene 
considered,  then  the  unfactored  s 
order  to  generate  the  reduced 
applied  twice.  For  the  R LNT  tabl 
times . 

In  summary,  the  factoriza 
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are  ineffective.  Possibly  ether 
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factorized  tables.  We  leave 
improved  assignments. 
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There  are  further  space  optim 
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example,  in  the  chain-reduced 
assignment  process  there  were 
entries  in  the  transition 
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ahead/apply  transition.  These  seventeen  transitions  may  be 
encoded  in  five  bits  in  the  transition  matrix  rather  than  the 
eight  bit  byte  suggested.  The  net  savings  in  table  space 
would  be  2G9  bytes. 

2)  Use  a  better  merger  algorithm  to  overlap  more  apply 
production  sublists.  The  method  employed  overlapped  only 
identical  sublists.  If  a  better  method  such  as  a  ’’topmost 
subset”  process  had  been  used,  possibly  greater  overlap  would 
have  resulted.  For  the  XPL  apply  production  list,  using  the 
"topmost  subset”  match  algorithm  decreased  the  chain-reduced 
apply  list  by  19  entries. 

3)  Pack  more  densely  seme  of  the  auxilliary  information  required 
by  the  running  parser.  For  example,  since  the  longest 
production  length  in  XPL  is  four  symbols,  the  length  minus 
one  may  be  packed  into  two  bits  {the  length  minus  one  is  the 
figure  needed  for  the  most  efficient  parse  stack  pop  after  a 
reduction) .  In  addition,  since  there  are  49  non-terminal 
symbols,  the  LHS  production  non-terminal  symbol  index  may  be 
packed  into  six  bits.  Thus  these  two  quantities  may  be 
packed  into  a  one  byte  entry  in  the  parsing  tables,  for  a 
savings  of  109  bytes.  Unfortunately  this  data  packing  fails 
for  SUSIE,  since  the  longest  production  length  is  eight. 
However,  there  are  only  25  non-terminal  symbols;  thus  the 
length  minus  one  may  be  encoded  in  three  bits,  and  the  LHS 
non-terminal  symbol  index  may  be  packed  into  five  bits.  Thus 
the  two  quantities  may  be  packed  into  a  one  byte  entry. 
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Optimizations  such  as  1)  and  3)  are  very  data  dependent,  and  as 
such  they  should  not  generally  be  employed  unless  minimal  space 
machines  are  desired,  and  the  data  accessing  algorithms  for 


parsing  may 

be 

modified 

to  suit 

the 

data  structures. 

In 
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overhead 

involved 

in 
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data 
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access  is  often  prohibitively  high.  Optimization  2)  would  be 
worthwhile  to  implement  in  a  production  program  since  it  is  a 
straightforward  procedure. 

In  concluding  this  section  on  table  space  results,  consider 
all  the  previous  optimizations  applied  to  the  tables  of  XPL.  It 
is  estimated  that  a  wcrhing  parser  with  full  error  detection 
could  be  generated  with  parser  tables  of  approximately  1300 
bytes.  If  error  detection  were  discarded,  the  parser  tables 
would  be  approximately  1000  bytes.  Thus  the  proposed 
optimizations  are  effective  in  achieving  table  space  reductions, 
but  compressing  the  tables  in  these  ways  will  inevitably  increase 
the  amount  of  code  in  the  parser  algorithm.  Without  these 
optimizations,  the  code  size  of  the  parser  is  roughly  the  same 
for  all  the  techniques  considered. 

Parsing  Speed  Considerations 

The  previous  sections  have  concentrated 
optimizations  of  parser  flow  tables.  In  this  section 
consider  the  speed  of  operation  of  the  reduced  matri 
tables,  and  show  why  they  parse  more  rapidly  than  list 
parsers. 
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,  the  matrix  structured  tables  may  he  accessed  by  directly 
ing  into  a  particular  table  storage  location  based  on  the 
es  of  the  current  parse  state  and  input  symbol. 


We  compare  the  list  lookup  method  with  that  of  th 
direct  index  lookup  method.  Assume  two  parser  models  i 
in  all  respects  except  that  one  employs  a  matrix  table  s 
and  the  other  a  list  structure.  Thus  insofar  as  parsing 
are  concerned  the  cnly  differences  between-  the  two  parser 
attributed  directly  to  the  differences  in  data  access 
figures. 


e  matrix 
dent ical 
tructure 
speeds 
s  can  be 
timing 


List  Search 

The  list  search  employed  typically  is  the  sequential  lookup. 
While  other  techniques  of  searching  could  be  used,  the  relatively 
short  list  length  found  in  compacted  list  parse  tables,  combined 
with  its  low  overhead  makes  it  a  logical  choice.  In  addition, 
the  sequential  list  search  is  the  technique  employed  by  Lalcnde 
in  the  LALR  parser  table  implementation  against  which  the  matrix 
lookup  method  will  be  compared. 
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We  make  an  assumption  to  simplify  the  list  search  analysis; 
The  probability  that  any  one  transition  from 
a  parse  state  will  be  followed  is  the  same  as 
that  for  any  ether  machine  transition. 

If  the  number  of  parse  states  in  the  parsing  machine  is 
Estates",  then  the  average  number  of  lookup  steps  per  parse  step 
is  the  average  number  of  transitiens  per  parse  state: 

#lcokups  =  # transit iens/ (2  *  fstates)  . 

The  evaluation  of  the  average  number  of  list  search 
iterations  has  been  done  for  three  different  implementations  of 
list  structured  parsers  for  the  XPL  grammar: 

1)  The  unoptimized  SLP(1)  parser  produced  from  DePemer* s 
analyzer ; 


2) 

The 

optimized  parser 

produced 

by 

Lalcnde ’ 

s  LALP; 

3) 

The 

optimized  parser 

produced 

by 

Anderson 

[  1972  ]. 

The  results 

are  given  in 

Table  4. 

20. 

Those 

for  LA LB  do  not 

include  the  number  of  productions  in  XPL  in  the  state  count,  so 
that  only  non-apply  states  are  considered  in  the  determination. 


The  estimate  of  the  average  number  of  list  search  iterations 
per  step  was  probably  too  low  for  LALR  because  the  lock-ahead 
states  were  counted  as  separate  states  in  the  calculation  of 


#states. 

whereas 

in  DeBemer’s 

SLB  ( 1 )  tables 

they  were 

no  t . 

The 

estimate 

based 

cn  Anderson 

*s  tables  was 

probably 

the 

most 

realistic 

of  the 

three . 
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Using  an  average  figure  of  two  and  one  half  iterations  per 
successful  list  search,  consider  the  average  execution  time  T 
required  for  each  search,  using  a  code  segment  cf  XPL  code 
extracted  from  the  LALB  parser: 

DO  I  -  1  TO  N ; 

IF  LIST  (I)  =  TOKEN  THEN  GO  TO  LOOP_END; 


END; 

LOOP_END:  ; 

At  the  successful  termination 
symbol  has  been  matched  to  the  LI 
a  basis  the  timing  figures  f 
[IBM],  the  machine  code  for  the  a 
approximately  23  microseconds 
usee,  for  a  successful  lookup. 


of  the  search  loop,  the  TOKEN 
ST  entry  indexed  by  I.  Using  as 
cr  the  IBM  360  model  44  computer 
bove  loop  was  found  to  require 
per  iteration  of  time  T,  or  57.5 


Matrix  Indexed  Lookup 

The  evaluation  of  the  time  for  a  matrix  entry  lookup  is 
dependent  upon  the  transition  type:  if  the  transition  is  to  a 
ncn-apply  state  the  XPL  code 

GO__TO_STATE  =  GO_TO  (HOW  (STATE)  , COL  (SYMEOi)  )  ; 
produces  the  destination  go-to  state  number  from  the  unassigned 
reduced  parser  tables,  where  SON  and  COL  are  the  indirect  address 
vectors  for  the  rows  and  the  columns  of  the  matrix  respectively. 
To  avoid  the  implicit  multiplication  involved  in  the  indexing 
calculation,  the  following  equivalent  code  was  used  instead: 

GO_TO_STATE  =  GO_TO  (SON  (STATE)  +  CCL  (SYMBOL)  )  ; 
where  the  SOW  vector  contained  the  row  indirect  address  entries 
premultiplied  by  the  number  of  matrix  columns.  Using  the  machine 
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the  above  XPL  code  segment 


instruction  timings  for  the  360/44, 
required  approximately  21.74  usee,  to  execute. 


The  following  code  segment  was  used  to  determine  the  parse 
restart  state  to  avoid  re-reading  the  produced  non-terminal  after 
a  production  is  applied: 

GO_TO_STATE  =  GO_TO  (ROW  (STATE_STACK  (ME)  )  , 

CCI  (LEFT_PART  (STATE)  )  )  ; 

where  STATE_STACK  (MP)  contains  the  stacked  parse  state  for 
resumption  of  the  parse,  and  MP  is  the  parse  stack  pointer; 
LEFT_PART  (STATE)  contains  the  production  LHS  symbol  produced 
after  production  STATE  is  applied.  ROW  and  CGL  are  the  row  and 
column  indirect  address  vectors  respectively  for  the  unassigned 
reduced  matrix.  The  same  optimization  employed  above  to  avoid 
the  indexing  multiplication  is  used  here: 

GO_TO_STATE  =  GO_TO  ( ROW  ( STATE_STACK  (M E) )  + 

COL  (LFFT_PART  (STATE)  )  )  ; 

for  a  total  calculated  execution  time  of  29.61  usee. 


Lookup  Comparison  and  Discussion 


The  timing  figures  estimated  for  the  execution  of  a  typical 
matrix  lookup  in  a  reduced  matrix  structure  parser  were 
calculated  to  be  approximately  half  that  of  the  corresponding 
list  search  lookup.  It  is  clear,  however,  that  these  figures  are 
indications  cnly  of  the  potential  benefits  of  using  a  matrix 
lookup  rather  than  a  list  lookup.  They  are  dependent  upon  the 
instruction  timings  for  a  particular  computer  executing  specific 
algorithms,  and  the  particular  set  of  tables  used  in  the 
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derivation  of  the  average  lookup  figure.  A  tetter  indication  of 
the  relative  merits  of  list  and  matrix  organizations  is  the 
comparison  of  two  parsers  -  one  list  structure  and  one  matrix 
structure  -  parsing  identical  input  texts.  The  texts  chosen  for 
the  tests  were  the  XPL  source  programs  for  the  XPL  compiler  XCOM, 
and  that  for  the  Student  PL  compiler  SPLCCW.  These  were 
processed  by  the  LALR  list  parser  and  by  an  unassigned  reduced 
natrix  parser.  Both  implementations  parsed  from  an  in-core  tcken 
array  prepared  by  a  scanner,  so  that  the  overhead  associated  with 
the  token  preparation  and  input  operations  was  eliminated.  The 
statistics  for  the  XCO M  and  SPLCCM  parses  are  shown  in  Table 
4.21,  where  the  numbers  and  types  of  parse  steps  are  tabulated 
for  each  of  the  two  parsers. 
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Grammar 

# 

ft  Parse 

Average 

Analyzer 

Tra  nsit ions 

States 

#  Lookups 

DeRemer  ' s 

925 

101 

4.57 

Lalon  de ’ s 

511 

125 

2.05 

Anderson ’ s 

510 

99 

2.57 

Table  4.20  -  Average  number  of  Sequential  List  Probes 

per  Parse  Step  for  XPL  Tables,  by  Analyzer 


XPL 

Source  Text 

XCOM 

SPLCOM 

ft  tokens 

24639 

37220 

ft  Parse  Steps 


-Bead 

246  39 

37220 

-Look-ahead 

28240 

43  192 

-Apply 

66748 

994  18 

Total  ft  Parse  Steps 

119627 

179830 

ft  Seg*l  List  Probes 


-Read 

79921 

121413 

—Look-ahead 

63304 

94502 

-Apply 

174201 

258498 

Total  ft  List  Probes 

317426 

474413 

Av.  ft  Probes  per  Step 


-Bead 

-Look-ahead 

-Apply 

Total  Av .  ft  Probes/Step 

3.24 

2.24 
2.60 
2.67 

3.26 

2.13 

2.60 

2.65 

Table  4.21  {a}  -  LALR  Parser 

Statist ics 

XPL 

XCOM 

Source  Text 

SPLCOM 

ft  Parse  Steps 

-Bead 

24638 

37219 

-Look-ahead 

24840 

38C71 

-Apply 

66748 

99418 

Total  ft  Parse  Steps 

116226 

174708 

Table  4.21  (b)  -  Reduced  Matrix  Parser  Statistics 
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From  the  results  in  Table  4.21,  the  estimate  of  two  and  one 

half  iterations  per  parse  step  was  reasonable.  The  actual 

< 

figures  produced  by  the  LALR  parse  were  2.67  and  2.65  for  XCOM 
and  SPLCOM  respectively.  Thus  the  simplifying  assumption  made  in 
the  derivation  of  this  estimate  was  not  too  unrealistic. 

Rased  on  the  statistics  of  the  XCC.M  and  SPLCCM  parses,  an 
approximate  calculation  may  be  done  to  determine  the  relative 
speed  improvement  of  a  matrix  parser  over  a  list  parser. 
Excluding  the  rest  of  the  parse  algorithms,  consider  only  the 
list  searching  code  component  of  the  total  parse  time  for  LALR. 
For  the  matrix  parser,  consider  the  indexing  time  for  the  apply 
production  steps,  the  read-lookahead  steps,  and  the  time  required 
to  check  the  error  matrix  for  the  presence  of  a  valid 
state/symbol  pair.  (The  error  check  XPL  code  was  calculated  to 
require  35.62  usee.)  The  total  list  search  time  approximation  is: 
List  time  =  av.  time  per  parse  step  *  Sparse  steps 
=  (23.0  *  2.65)  *  119627  usee. 

=  7.29  sec . 

The  total  matrix  parse  lookup  and  error  check  time  is: 

Matrix  time  =  apply  time  +  read-LA  time  +  error  check  time 

=  (66748*29 .6  1)  +  (24 639+ 2  824 0)  *2  1 . 7  4 

+  (246  39*35. 62)  usee. 

=  4.00  sec. 

Therefore  the  relative  improvement  of  the  matrix  parser  above  the 
LALR  parser  parsing  the  XCOM  source  is  7.29/4.00  or  1.82.  (it 
has  been  assumed  that  the  parse  time  for  a  lcok-ahead  step  is  the 
same  as  that  for  a  read  step.  This  is  an  approximation,  since 
much  less  code  is  executed  for  a  lock-ahead  than  for  a  read.) 
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Osing  the  predicted  estimate  cf  2.5  iterations  per  successful 
list  lookup,  the  predicted  improvement  ratio  is  1.72. 

The  timing  figures  for  the  parser  executions,  shown  in  Table 
4.22,  also  include  those  for  the  execution  cf  the  MSP  parser  for 
comparison.  All  timing  figures,  obtained  from  executions  in  a 
quiesced  360/44,  are  elapsed  timings  in  seconds.  These  results 
show  that  the  reduced  matrix  parser  with  full  error  detection 
capability  parsed  at  a  rate  1,7  times  faster  than  LALR  and  2.8 
times  faster  than  MSP.  The  actual  improvement  in  the  matrix 
parser  speed  over  LALR  is  very  close  to  the  predicted  value. 
With  no  error  detection,  the  reduced  matrix  parser  operated 
nearly  1.9  times  faster  than  LALR  and  3.1  times  faster  than  MSP. 

The  reduced  error  detection  matrix  E*  has  been  described  as 
being  implemented  in  a  bit  matrix,  packed  eight  bits  per  byte.  A 
mention  was  made  that  the  bit  lookup  necessitated  by  this  packing 
implied  that  some  parsing  speed  would  be  lost  because  of  the  time 
required  for  error  checking.  While  seme  speed  is  lost  due  to 
this  check  (35.62  micr oseconds  per  input  symbol,  since  the  check 
is  performed  only  once  per  input  token),  the  overhead  for  this 
checking  requires  only  10.4%  of  the  total  parse  time.  This 
overhead  could  be  reduced  by  eliminating  the  dense  bit  packing  of 
the  matrix,  but  the  overhead  is  small  when  weighed  against  the 
large  matrix  storage  space  required  for  an  unpacked  error  table. 

The  parser  statistics  of  Table  4.21  contain  some  interesting 
results  indirectly  related  to  speed  considerations.  Aside  from 
the  fact  that  the  LALR  parser  reads  one  mere  token  than  the 
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matrix  parser  (the  goal  post  symbol  is  not  read  by  the  matrix 
parser)  ,  the  number  of  lock-ahead  parse  steps  differs 
considerably  between  the  two  parsers:  IALR  performs  3400  more 


look-aheads  parsing  XCCM  and 
matrix  parser.  The  differences 
has  separated  all  look-ahe 
transitions  in  the  machine,  and 
ahead  states.  If  a  symbol 
contained  look-aheads  before  sep 
to  the  look-ahead  state  mach 
Control  is  then  passed  to  the  re 
This  extra  overhead  is  avoided 
maintaining  both  read  and  look-a 
parse  states  as  they  were  origin 
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Reassigned  Reduced  Matrix 


Parser  Speed 


Although  matrix  tables  were  generated  for  the  reassigned 
reduced  parser  fcr  the  XPL  grammar,  test  results  were  not 
obtained  by  running  a  suitable  parser.  The  execution  timings 
required  to  parse  XCOM  and  SPLCOrl  must  thus  be  determined  by 
examining  the  XPL  code  segments  required  to  decode  the  table 
entries. 

To  convert  the  table  entry  numbers  in  the  reassigned  reduced 
machine  to  absolute  state  numbers,  assume  that  each  entry 


occupies  one  byte. 

and 

that 

the  ’’apply/nc  apply” 

flag 

occu pie 

the  high-order  bit  in 

the 

byte. 

If  the 

high -order 

bit 

is  ”1” 

then  the  entry  ref 

ers 

to  an 

’’apply  ” 

transition; 

if  it 

is  "0” 
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then  it  refers  to  a  "no-applyM  transition,  A  suitable  XPL  code 
segment  to  determine  the  transition  type  is: 

DATA  =  GO_TO  (ROW  (STATE)  +  COL  (SYMBOL)  )  ;  /*  fetch  entry  */ 

If  DATA  >  ’’BO”  THE!! 

/*  an  apply  transition  */ 

STATE  =  APPLY_TABLE  (APPL Y_TAELE_PTR  (STATE)  + 

(DATA  Z  ”7F”)  )  ; 

ELSE 

/*  a  ncn-apply  transition  */ 

STATE  =  EASE_TABLE (SYMBOL)  +  DATA; 

where  STATE  is  the  absolute  state  number, 

SYMBOL  is  the  absolute  input  symbol  vocabulary  index, 
APPLY_TAELE  is  the  production  number  data  list, 
APPLY_TAELE_PTR  (STATE)  is  the  pointer  to  the  start  of 
apply  sublist  for  STATE, 

EASE_TABLE  (SYMBOL)  is  the  ind irec tic n ■  b ase  address 
entry  for  the  input  SYMBOL. 

The  above  code,  which  must  be  used  after  each  matrix  fetch, 
increased  the  loop  execution  timings: 

Time  increase  for  each  apply  transition  :  21.20  usee. 

Time  increase  for  each  non-apply  trans’n:  14.75  usee. 

The  total  elapsed  parse  timings  were  thus  IB. 55  seconds  for 
XCOM  and  27.85  seconds  for  SPLCOM  using  the  reassigned  reduced 

matrix  parser  with  full  error  detection  capabilities.  This 

parser  could  process  XPL  tokens  approximately  1.5  times  faster 
than  LAIR,  and  nearly  2.5  times  faster  than  MSP, 
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Chain  Rule  Elimination  Results 


As  previously  mentioned  in  chapter  4,  the  modified  Aho  and 
Ullman  algorithm  was  employed  in  an  effort  to  eliminate  chain 
productions  from  XPL,  Eleven  semanticless  productions,  as 
determined  from  the  compiler  source  cede,  were  chosen  for 
elimination,  with  seven  of  them  in  the  derivation  of 
<expression>.  Unfortunately  the  production 

<logical  secondary>  <logical  primary> 

in  the  middle  of  the  expression  derivation  could  not  be  chosen 
for  elimination  because  of  its  associated  semantics.  Had  there 
been  no  semantics  for  this  production  the  entire  chain  rule 
der iva  tion 

<expression>  <logica.l  factor>  :  : =  ...  :  :=  <constant> 

would  have  been  candidates  for  elimination.  However,  because  of 
the  absence  of  the  LI?  (0)  reduce  states  in  the  non-canonical 
tables  processed,  cne  rule  -  production  80 

<logical  primary>  <string  expression> 

could  not  be  eliminated.  This  was  detected  in  step  4  of  the 
modified  algorithm. 

The  parsing  statistics  for  the  ten  chain  rule  eliminated  XPL 
parser  are  shown  in  Table  4.23,  and  the  timing  figures  in  Table 
4.24. 
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Source 

Text  Parsed 

Parser 

XCOM. 

SPLCOM 

MSP 

45.51 

67.02 

1ALR 

27.56 

4  1.25 

Reduced 

Matrix 

-full 

error  det’n 

16.40 

24.6  3 

-nc  error  det ' n 

14.70 

22.08 

Table  4.22  -  Parser  Timing  Tests 

(elapsed  time  in  seconds) 


Source  Text  Parsed 


XCOM 

SPICCM 

#  Parse  Steps 

-Read 

24638 

37219 

-Lcck- ahea  d 

10433 

17449 

-Apply 

39370 

60103 

Total  #  Steps 

74  44  1 

114771 

Table  4.23  -  Chain  Production  Reduced 

Matrix  Parser  Statistics 


Source  Text 

Parsed 

Parser 

XCOM 

SPLCOM 

Reduced 

Matrix 

-full 

error  det’n 

11.06 

16.92 

-no  error  detTn 

9.37 

14.35 

Table  4 . 24  -  Chain  Rule  Reduced  Matrix  Parser 

Timing  (elapsed  time  in  seconds) 


State 

Chain  Rule 

Frrcr 

Normalized 

Parser 

Assigned 

Re  duced 

Det 1  n 

Parse  Time 

MSP 

— _ 

4.13 

LALR 

— 

— 

— 

2.50 

Reduced  Matrix 

no 

no 

yes 

1.48 

n 

no 

no 

no 

1.3  3 

t« 

ves 

u. 

no 

yes 

1.68 

it 

yes 

no 

no 

1.53 

it 

no 

yes 

yes 

1.00 

it 

no 

yes 

no 

C  .  8  5 

it 

yes 

yes 

yes 

1.13 

ti 

yes 

yes 

no 

0.97 

Table  4.25  -  Normalized  Parse  Time  of  XPL  Source  Text 
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The  parser  timing  figures  show  the  effectiveness  of  chain 
production  elimination:  a  matrix  parser  was  produced  that 
operated  about  1.5  times  faster  than  the  same  parser  without 
chain  elimination.  Furthermore,  this  matrix  parser  with  chains 
eliminated  parsed  at  a  rate  2.5  times  faster  than  the  LALR  parser 
and  4  times  faster  than  the  MSP  parser.  Kith  the  error  detection 
capability  eliminated,  the  chain  eliminated  matrix  parser 
operated  almost  3  times  faster  than  the  LALR  parser,  and  nearly  5 
times  faster  than  the  MSP  parser. 

It  is  apparent  that  the  elimination  of  chain  rules  enabled  a 
significant  parser  speedup.  This  speedup  is  applicable  both  to 
list  structured  parser  tables  and  to  matrix  tables  as  well. 
However,  it  is  likely  that  list  structured  tables  would  not 
achieve  as  great  a  speed  improvement  as  that  observed  in  chain 
eliminated  matrix  tables  when  Aho  and  Ullman’s  algorithm  is  used. 
This  would  not  be  the  case  in  Anderson’s  chain  eliminated  tables. 
As  pointed  out  by  Aho  and  Ullman  [1972b],  their  chain  rule 
elimination  algorithm  tends  to  increase  the  number  of  transition 
entries  in  machine  parsing  states.  This  would  have  the  effect  of 
increasing  the  list  lookup  time  required  fcr  a  successful  search. 
No  such  time  increase  would  be  noticed  in  a  matrix  structure 
because  the  lockup  time  is  not  dependent  upon  the  number  of 
transitions  per  state,  nor  on  parser  table  size.  Thus  parsing 
speed  would  not  suffer  in  a  matrix  parser  because  of  chain  rule 
elimination . 

The  elimination  of  the  ten  chain  rules  was  effective  in 
reducing  the  number  of  productions  applied.  The  relative  percent 
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reduction  could  have  been  estimated  frcrr  Alexander’s  results 
[1972],  where  he  showed  that  the'  eliminated  productions 
represented  approximately  34.7%  of  the  total  number  of 
productions  executed  in  a  sample  of  XPL  source  programs.  Thus 
65.3%  of  the  reductions  will  still  be  performed  after  chain 
elimination.  From  the  results  in  table  4.23,  59%  and  61%  of  the 
original  numbers  of  productions  were  applied  in  the  parse  of  XCGM 
and  SFLCOM  respectively. 

As  an  extra  benefit  of  chain  rule  elimination,  the  number  of 
look-ahead  steps  required  in  parsing  the  two  XPL  programs  was 
significantly  reduced:  for  the  XCCI1  parse  to  36% ,  and  for  the 
SPLCOtf  parse  to  40%  of  the  number  of  the  number  of  look-aheads  in 
the  LALR  parse.  This  reduction  has  cccured  because  many  of  the 
eliminated  productions  were  accessed  only  thccugh  lock-ahead 
transitions.  The  elimination  of  chain  rules  has  thus  reduced  the 
total  number  of  parse  steps  to  54%  and  .66%  of  the  numbers 
required  by  the  matrix  parser  without  chain  rules  eliminated. 

Reassigned  Chain  Buie  Reduced  Matrix  Parser  Speed 

As  was  previously  discussed,  the  parsing  speed  of  the 
reassigned  reduced  matrix  parser  must  be  calculated.  Using  the 
chain  rule  eliminated  tables  for  the  parser,  the  net  calculated 
parse  time  for  XCCM  would  be  12.4  1  seconds,  and  that  for  SP.LCOM 
would  be  19.00  seconds.  These  timings  show  the  chain  reduced 
assigned  matrix  parser  operates  2.2  times  faster  than  the  LALR 
parser,  and  3.6  times  faster  then  the  PiSP  parser. 


-109- 


. 


* 


Comparison  with  Published  Results 

< 

Anderson  [1972]  has  provided  results  of  speed  tests  for 
several  different  chain  reduced  parser  variations  for  the  syntax 
analysis  phase  of  the  Algol  W  language  compiler,  and  has  compared 
these  to  the  highly  efficient  simple  precedence  analyzer  in  the 
Algol  ft  compiler.  This  simple  precedence  parser,  designed  and 
written  by  S.  L.  Graham,  contained  an  unpublished  mechanism  for 
bypassing  chain  reductions.  The  results  of  Anderson’s  speed 
tests  have  demonstrated  for  the  sample  Algol  ft  programs  tested, 
that  the  Algol  ft  parser  and  tie  chain  reduced  SLR  parser  he 
employed  were  equivalent  in  parsing  speeds.  His  results  showed 
that  these  syntax  analyzers  operated  at  a  cate  approximately  1.5 
times  that  of  the  SLR  and  simple  precedence  analyzers  without 
chains  bypassed.  fte  also  have  found  in  our  study  a  similar  order 
of  speedup  using  a  chain  reduced  matrix  parser. 

Anderson  has  achieved  this  figure  of  a  1.5  times  speedup 
partly  because  he  employed  the  list  searching  hardware  of  the 
/360  computer.  Seme  of  this  speedup  would  have  been  lost  had  he 
employed  a  sequential  list  search  implemented  by  programming 
rather  than  special  hardware  instructions.  Our  results  were 
achieved  without  benefit  cf  specialized  hardware  features.  As 
such,  our  techniques  of  reduction  of  parser  matrix  structures  are 
mere  generally  applicable  than  are  Andersen’s. 

In  summary,  we  have  demonstrated  that  the  chain  rule  reduced 
matrix  parser  is  significantly  faster  than  or  as  fast  as 
previously  published  results.  The  normalized  speed  comparisons 
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aue  presented  in  Table  4,25,  using  the  XCC  M  parse  time  by  the 

chain  rule  reduced  parser  with  full  error  detection  as  the 

< 

normal, 

A  Practical  .Application 

Lalonde  [1971]  has  stated  that  in  the  XPL  compiler,  the  MSP 
parser  required  19%  and  the  LALR  parser  11%  of  the  total 
compilation  time  for  typical  XPL  programs,  Ey  using  the  chain 
production  eliminated  reduced  matrix  parser  instead,  the  fraction 
of  the  total  execution  time  spent  in  parsing  drops  to 
approximately  4%,  Thus  a  15%  saving  in  compilation  time  may  be 
achieved  by  using  the  matrix  parser  rather  than  the  MSP  parser 
employed  in  the  distribution  version  of  the  XPL  compiler  XCOM. 
In  Appendix  V,  we  have  calculated  the  net  savings  expected  by 
using  a  matrix  parser  in  this  compiler  to  be  approximately  $2.00 
per  compilation  CPU  minute  {excluding  I/O  charges)  at  the 
University  of  Toronto  Computer  Centre,  using  the  installation’s 
IBM  370  Model  165. 
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Cha n t cr_5 

Lexical  Analysis 

It  is  evident  from  the  results  of  Chapter  4  that  by  employing 
a  matrix  parser  in  place  of  the  MSP  or  JLALR  parsers  in  the  XPL 
compiler  that  the  total  parse  time  required  by  the  compiler  is 
reduced  to  a  level  where  further  parser  improvements  would  cause 
little  change  in  the  total  compilation  time.  The  matrix  parser 
and  tables  have  effectively  removed  parsing  as  a  speed  bottleneck 
in  the  compilation  process.  It  thus  becomes  practical  to 
consider  the  application  of  parser  generating  techniques  to  the 
area  of  lexical  analyzers.  The  power  and  flexibility  of  general 
LK  (k)  grammar  parsers  are  not  always  required  or  needed  in 
lexical  analyzers,  even  though  the  operation  of  a  scanner  Is 
essentially  the  same  as  that  of  a  parser. 

If  parsing  techniques  can  be  applied  in  an  organized  fashion 
to  the  automatic  generation  of  lexical  analyzers,  there  would  be 
three  immediate  benefits: 

1)  Simplified  construction  of  scanners  in  the  compiler 
generation  process; 

2)  Elimination  of  the  arbitrary  distinction  between  a 
scanner  and  a  parser; 

3)  Unification  of  the  approach  to  constructing 
lexical-syntactical  "front  ends”  for  compilers. 

The  construction  of  lexical  analyzers  would  be  simplified  In 
the  compiler  generation  piocess  by  specifying  the  syntax  of  the 
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terminal  language  token  strings  for  a  language,  and  the  semantic 

actions  to  be  performed  on  token  constituents.  For  example,  BNF 

< 

notation  could  le  employed  to  specify  the  makup  of  terminal 
tokens.  This  BNF  could  be  processed  by  an  analyzer  which  would 
produce  scan  decision  tables  from  the  BNF.  Semantic  actions 
could  be  associated  with  each  syntactic  production  of  the  BNF  and 
could  be  invoked  as  each  production  were  applied  during  lexical 
analysis  of  terminal  strings.  In  essence,  the  same  processes 
used  in  generating  a  programming  language  parser  could  be  used  to 
generate  a  programming  language  scanner,  thus  unifying  the 
approach  to  lexical  and  syntactic  processing  of  Input  text. 

One  of  the  major  reasons  for  the  distinction  that  has  been 
made  between  the  syntactic  and  lexical  phases  of  the  compilation 
process  has  been  the  volume  of  data  processed  by  each  phase.  The 
lexical  analyzer  typically  operates  on  input  data  streams  that 
are  several  orders  of  magnitude  larger  than  that  of  the  syntactic 
analyzer.  Thus  for  the  sake  of  speed  of  the  overall  compilation 
process  the  lexical  analyzer  should  be  as  highly  efficient  as 
possible. 

In  this  chapter,  we  will  show  that  by  using  formal  methods  it 
will  be  possible  to  generate  a  lexical  analyzer  that  is 
competitive  in  speed  with  a  hand-coded  scanner.  By  illustrating 
the  feasibility  of  this  concept,  the  distinction  between  scanners 
and  parsers  may  be  eliminated.  Theoretically  the  syntactic 
structure  of  the  terminal  tokens  may  be  included  in  a  grammar  for 
the  entire  language,  rather  than  In  a  separate  supposedly  more 
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efficient  scanner.  The  syntactic  and  lexical  phases  would  then 
become  a  single  phase  of  a  compiler. 

< 

lexical  -Analysis  of  XPL 

To  illustrate  the  formal  construction  technique  applied  to 
the  automatic  generation  of  scanners,  the  terminal  token 
structure  for  the  XPL  language  was  determined  and  encoded  into  a 
62  production  grammar.  This  grammar  included  the  token  classes 
for  XPL  identifiers,  constants,  delimiters,  character  strings, 
bit  strings,  comments,  etc.,  and  is  given  in  Appendix  VI,  Rather 
than  include  the  entire  XPL  grammar  in  the  scanner  grammar,  the 
first  seventeen  productions  provide  a  replacement  for  it.  These 
productions  specify  the  set  of  delimiters  for  each  token  class, 
and  drive  the  lexical  analysis  onward  after  the  recognition  of 
each  token.  The  grammar  in  Appendix  VI  is  not  to  be  interpreted 
as  the  ultimate  in  lexical  grammar  design..  It  is  a  working 
grammar  specif icat ion . 

One  of  the  problems  of  processing  such  a  grammar  is  the 
presence  of  symbols  such  as  the  XPL  comment  delimiters  */**  and 
1  V1*  These  symbol  pairs  both  reguire  look-aheads  greater  than 
one  in  order  to  unambiguously  classify  them.  Since  DeRemer’s 
SLR ( 1 )  analyzer  was  to  be  used  to  process  the  scanner  grammar, 
these  symbols  were  checked  for  using  local  lookahead  in  a  scanner 
for  the  lexical  analyzer  (ie-  a  "scanner-scanner"),  and  thus 
classified.  Another  problem  -  the  recognition  of  grammar 
keywords  -  was  handled  exactly  as  in  the  XPL  hand-coded  scanner. 
When  an  identifier  had  been  collected,  it  was  compared  against 
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the  list  of  valid  keywords  for  XPL.  If  a  match  was  found,  the 

token  was  classified  as  a  keyword  not-  as  an  identifier.  The 

< 

recognition  of  macro  usage  was  handled  in  a  manner  identical  to 
that  of  keywords.  The  recognition  and  handling  cf  other  such 
context-sensitive  conditions  was  also  handled  by  the  "scanner- 
scanner"  . 

The  scanner  grammar  was  processed  by  the  SLR(1)  analyzer, 
then  by  the  parser  matrix  reduction  program.  The  tables  emitted 
were  inserted  into  the  reduced  matrix  parser  framework.  Speed 
tests  were  then  performed  on  the  formal  scanner  and  on  the  hand¬ 
written  XCOM  scanner  for  XPL.  As  in  previous  speed  tests,  the 
test  programs  were  the  XPL  source  of  XCGM  and  SPLCOM. 

Results  of  Tests 

The  statistics  of  the  scanner  tests  are  given  in  Table  5.1. 
The  handwritten  XCCM  scanner  processed  text  at  nearly  2900  cards 
per  minute  (elapsed)  on  the  360/44,  whereas  the  formal  fully 
syntax  directed  matrix  scanner  processed  text  at  only  1150  cards 
per  minute  scanning  XCOM.  The  results  obtained  scanning  SPLCOM 
showed  an  even  larger  difference:  3406  cpm  for  the  handwritten 
scanner  against  1191  cpm  for  the  formal  scanner.  These  figures 
seem  to  indicate  that  the  formal  scanner  is  not  competitive  from 
the  scan  rate  viewpoint. 

In  an  attempt  to  improve  the  scan  rate  of  the  formal  scanner, 
nineteen  chain  productions  were  eliminated  from  the  scanner 
tables  by  the  modified  chain  elimination  algorithm.  Shen  the 
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speed  tests  were  performed  again,  the  scan  rate  improved  slightly 
more  than  200  cpm.  The  speedup  benefit  from  the  chain  production 
elimination  was  not  as  much  as  had  been  anticipated  from  the 
parser  test  results  of  a  1.5  times  improvement.  However,  the 
number  of  productions  applied  was  cut  to  55%  of  the  number 
applied  without  chains  eliminated.  The  number  of  look-ahead 
parse  steps  was  also  decreased  slightly  as  a  consequence  of  the 
chain  rule  elimination,  but  not  as  much  as  was  anticipated  from 
the  decrease  in  lock-aheads  when  chain  reduction  was  applied  to 
the  XPL  parser  tables. 

Scanner  Modification 

Alexander  [1572]  has  shown  that  11.2%  of  the  total 
compilation  time  of  the  XCCM  compiler  was  involved  with  the 
scanning  of  blanks  in  typical  XPL  programs.  Ey  replacing  the 
blank  scanning  Icop  in  the  XCCM  hand-written  scanner  with  a  more 
efficient  t r ansla te-and- test  (TFT)  instruction,  Eobas  [1972]  was 
able  to  reduce  this  figure  to  0.7%. 

The  increase  in  scanning  efficiency  obtained  by  replacing  the 
blank  scanning  logic  in  the  XCOM  scanner  suggested  that  this  was 
a  suitable  place  to  optimize  the  syntax-directed  scanner. 
Consequently  the  formal  scanner  was  modified  to  perform  blank 
scanning  in  a  code  loop  that  was  identical  to  the  blank  scan  loop 
employed  in  XCCM.  The  scanning  speed  tests  were  repeated  with 
gratifying  results:  the  scan  rate  increased  to  2496  cpm  scanning 
XCOM ,  and  to  2689  cpm  scanning  SPLCCM,  for  a  speed  gain  of  85% 
above  the  chain  reduced  scanner.  Equally  significant  was  the 
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drop  in  the  number  of  read  and  apply  parse  steps,  while  the 
number  of  look-ahead  steps  was  unchanged.  The  magnitude  of  the 
reduction  in  each  of  the  read  and  apply  steps  could  have  been 
predicted,  since  Alexander  had  found  that  about  681?  of  most  XPL 
programs  were  blanks.  Thus  eliminating  the  syntax-directed 
scanning  of  blanks  would  eliminate  two-thirds  of  the  number  of 
read  steps  and  two-thirds  of  the  number  of  apply  steps. 

Alexander  also  had  stated  that  about  145?  of  the  typical  XPL 
program  was  within  comment  delimiters,  with  the  XCCM  compiler’s 
comment  content  being  11%.  Thus  a  further  reduction  in  the 
number  of  read  and  apply  parse  steps  could  be  achieved  with  a 
resulting  scanner  speed  gain  by  not  syntactically  analyzing 
comments.  The  syntax-directed  comment  scan  was  replaced  by  a 
loop  of  XPL  code  -  the  same  code  as  was  employed  in  XC0I1  to  scan 
comments,  and  the  scan  tests  were  repeated.  The  scan  rate  of  the 
modified  syntax-directed  parser  with  blank  and  comment  scan  loops 
increased  to  2929  cpm  scanning  XCCE  and  3284  cpin  scanning  SPLCCM. 
A  decrease  of  10%  in  the  number  of  read  parse  steps  and  1%  in  the 
number  of  apply  parse  steps  was  observed.  Again  Alexander’s 
empirical  results  were  employed  to  successfully  improve  the 
syntax-directed  scanner’s  speed. 

Further  speed  improvements  of  the  formal  scanner  could  be 
obtained  by  code  modifications  such  as  employed  by  Bobas  to  cut 
blank  scanning  overhead.  To  demonstrate  this  possibility,  the 
XCOM  blank  scan  loop  was  replaced  in  the  syntax-directed  scanner 
by  a  very  efficient  three  machine  instruction  blank  scan  loop, 
using  the  XPL  INLINE  code  capability,  and  the  speed  tests  were 
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loops  as  XCO M  scanned  at  approximately  the  same  rate  as  the 
scanner.  Thus,  we  have  demonstrated  that  the  generation  of 
xical  analyzer  by  formal  methods  and  its  subsequent 
ization  to  be  a  viable  technigue  for  scanner  generation. 
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was  greater  than  one.  The  same  was  true  for  the  scanning  of 
comments.  Thus  an  alternate  approach  was  adopted. 

Two  bottlenecks  were  recognized  in  the  scanning  process  - 
blank  scanning  and  comment  scanning  -  which  together  accounted 
for  the  majority  of  scanning  decisions.  Component  machines  were 
formed  from  the  main  scanning  machine  by  factoring.  These 
component  machines  for  blanks  and  comments  were  optimized  for 
speed,  and  then  recombined  with  the  main  scanning  machine.  While 
it  may  be  argued  that  the  use  of  encoded  loops  for  these  factored 
machines  is  not  formalized,  we  contend  that  in  principle  the  same 
results  could  have  been  achieved  if  a  locally  optimal  machine 
structure  could  have  been  formulated  for  them. 
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1)  Handwritten 

scanner 

in  XCCM 

compiler 

Scanner  2) 
Scanner  3) 
Scanner  4) 
Scanner  5) 
Scanner  6) 


(uses  blank. scan  lccp) 

Fully  syntax  directed  scanner 

Same  as  Scanner  2,  with  chain  rules  eliminated 

Same  as  Scanner  3,  with  blank  scan  code  loop 

Same  as  Scanner  4,  with  comment  scan  loop 

Same  as  Scanner  3,  with  fast  INLINE  blank  scan  loop 

and  comment  scan  loop 


Table  5 . 1 


Scanner  Test  Besults 


Source  Text 

Scanned 

Scanner + 

XCOM 

SFLCC W 

1) 

1.00 

1.00 

2) 

2.54 

2.87 

3) 

2.  14 

2.43 

4) 

1.16 

1.27 
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0.99 

1 .  C  4 

6) 

0.90 

0*93 

Table  5,2  -  Normalized  Scan  Time 

(  +  See  legend  Table  5*1) 
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Compiler  "Front  End”  Construction 

The  above  principle  may  be  extended  in  a  logical  way  to  cover 
the  production  of  a  very  efficient  compiler  "front  end”.  The 
entire  grammar  for  a  compiler  could  be  specified  to  an  analyzer, 
including  the  structure  of  the  terminal  toXen  set.  This  grammar 
would  be  analyzed,  and  portions  of  the  grammar  separated 
depending  upon  their  characteristics.  Each  grammar  portion  would 
be  processed,  and  scanning/parsing  decision  tables  produced  for 
each  that  would  be  speed  optimal.  Thus  portions  of  the  grammar 
that  (say)  were  regular  could  be  handled  by  a  simple  finite  state 
machine,  and  portions  that  reguired  a  stack  could  be  handled  by  a 
more  general  IT?  ( 0 )  or  SLE(1)  DPE£  parsing  algorithm.  Each 
portion  of  the  grammar  would  thus  use  only  as  much  of  the 
scanning/parsing  resources  required,  and  no  more  than  reguired, 
during  a  compilation.  Had  such  a  segmentation  algorithm  been 
available,  it  probably  would  have  generated  small  sequential 
recognition  machines  for  the  terminal  token  sets  of  XPL  since 
they  are  chiefly  regular  in  nature. 

To  simplify  the  generation  of  segmented  grammars,  the 
notation  used  to  specify  the  structure  of  the  language  to  be 
processed  could  include  a  form  of  regular  expression  notation,  as 
well  as  a  ENF  form.  The  regular  expressions  would  be  handled  by 
a  FSM  recognizer,  and  the  more  general  BNF  by  an  SLR  or  LR 
parser.  Undoubtedly  several  ether  different  notations  could  also 
be  employed  to  advantage.  However,  rather  than  following  these 
notions  further,  we  leave  them  for  future  consideration.  Even 
with  segmentation,  the  optimization  techniques  described 


-12  1- 


. 

' 


. 

previously  may  be  applied  successfully  to  improve  the  operating 
speed  of  each  segmented  machine  independently. 

< 

If  the  compiler  designer  wishes  to  maintain  a  logical 
distinction  between  the  scanning  and  parsing  phases  of  a 
compiler,  the  formal  approach  to  the  generation  of  both  the 
scanner  and  the  parser  may  still  be  used  to  advantage.  By 
employing  the  same  algorithm  in  the  generation  of  the  scanning 
and  parsing  tables,  the  designer  gains  the  full  povrer  of  high- 
level  ^SIR(I))  grammar  constructs  in  the  terminal  token  classes. 
Such  constructs  as  the  arbitrary  nesting  cf  declarations  or 
recursively  defined  tokens  may  be  readily  handled  by  a  DP DA 
scanner.  There  are  other  advantages  in  using  a  sy nt ax- directed 
scanner.  The  same  algorithm  that  drives  the  parser  may  be  used 
to  drive  the  scanner  by  switching  the  algorithms  stack  and 
tables.  This  saves  program  code  space  and  simplifies  the  scanner 
logic.  In  addition,  the  same  syntax-directed  errcr  recovery 
techniques  employed  for  parser  error  recovery  may  be  used  for 
scanner  error  recovery  as  well.  Eoth  of  these  help  to  simplify 
the  structure  of  a  compiler  by  reducing  the  unnecessary 
duplication  of  function  within  the  parser  and  scanner  framework. 
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Chapter  6 


Conclusions 


The  objectives  of  this  work  were  to  produce  practical  matrix 
parser  tables  for  LR (k)  grammars  whose  tabular  representations 
were  small  and  whose  parsing  rates  were  as  high  as  possible.  An 
integral  requirement  imposed  cn  the  parsing  table  structure  was 
that  its  error  detection  capability  must  be  identical  to  that  of 
the  original  LR(k)  grammar  finite  state  parser. 

The  error  detection  capability  of  an  LR  (k)  parser  was 
introduced  as  a  property  that  may  be  factored  from  the  parser 
table  matrix  structure.  The  development  of  successor  error 
equivalence  (SEE)  as  an  error  preserving  definition  of 
compatibility  enabled  the  LR (k)  EFA  to  he  reduced  by  formal 
techniques.  The  error  detection  capability  of  the  parser  was 
thus  retained  unaltered  in  a  reduced  matrix  structure. 

Disjoint  compatibility,  a  special  case  of  the  SEE 
compatibility  relation,  was  shown  to  be  equivalent  to  the  SEE 
relation  between  states  of  a  Deft  enter  LALR  (k)  parser.  Thus  the 
generation  of  a  suitable  reduced  closed  cover  of  the  LALR  (k) 
matrix  parsing  tables  was  considerably  simplified,  since  closure 
of  the  reduced  cover  state  set  of  the  machine  was  guaranteed. 

The  practicality  of  generating  reduced  matrix  parsing  tables 
has  been  verified  experimentally.  An  adaptation  of  the  reduction 
technique  introduced  by  Pauli  and  Unger  has  been  developed  using 
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the  method  of  Roy  and  Sheng  that  takes  advantage  of  disjoint 
compatibility.  This  algorithm  is  shewn  to  be  practical,  simple, 
and  non-enumera ti ve .  It  is  effective  in  producing  theoretically 
minimal-state  parsers  for  most  of  the  grammars  investigated.  Its 
time  complexity,  quadratic  in  the  number  of  parser  states,  makes 
the  minimization  of  LALR  (k)  parser  tables  of  several  hundred 
states  practical.  The  reduction  algorithm  has  also  been  applied 
effectively  to  the  reduction  of  the  number  of  columns  in  a  LALR 
parser  table,  as  well  as  to  row- then-column  and  column- then-row 
reductions. 

A  heuristic  state  assignment  procedure  has  been  designed  to 
increase  the  commonality  among  rows  and  among  columns  of  a  LALR 
parsing  matrix  table.  Its  application  prior  to  reduction  of  the 
table  resulted  in  a  significant  decrease  in  the  matrix  size  after 
the  reduction  cf  the  matrix. 

The  LR (k)  parser  matrix  reduction  of  Pager  [1970]  and  of  Aho 
and  Ullman  [1972]  have  been  shown  to  he  subsumed  both 
theoretically  and  practically  by  this  work. 

The  evaluation  of  the  implementation  sizes  of  reduced  matrix 
parsers  has  shown  them  to  be  competitive  in  size  with  the  tables 
produced  by  existing  parser  generators,  specifically  McKeeman's 
MSP  generator  and  Lalonde's  LALR  generator.  However,  the  speed 
of  operation  of  the  matrix  parser  was  considerable  better  than 
that  of  either  MSP  or  LALR  parsers. 
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It  is  evident  that  the  objective  of  achieving  a  high  parsing 

rate  has  been  realized  by  the  reduced-  matrix  parser.  In  fact, 

< 

substituting  a  reduced  matrix  parser  for  the  existing  parser  in 
the  XPL  compiler  reduced  the  parsing  time  to  four  percent  of  the 
total  compilation  time,  effectively  eliminating  the  time  spent 
parsing  as  a  major  factor  to  be  considered  in  the  compilation 
process . 

The  viability  of  the  reduced  matrix  table  structure  has  been 
further  demonstrated  by  applying  LR  (k)  generation  and  matrix 
reduction  techniques  to  lexical  analyzers.  Py  these  techniques, 
an  SLR ( 1 )  lexical  analyzer  for  XPL  has  teen  produced  that  scans 
at  the  same  rate  as  the  handwritten  scanner  in  the  XPL  compiler. 

In  summary,  our  design  objectives  have  been  met.  The 
representation  of  LR(k)  parser  tables  in  a  reduced  matrix  error¬ 
preserving  structure  was  practical  and  economic.  The  practical 
algorithms  and  transformations  introduced  have  enabled  the  rapid 
production  of  reduced  matrix  parser  tables.  The  implementation 
space  requirement  for  the  matrix  parser  tables  was  competitive 
with  existing  parser  representations.  The  parser  speed  tests 
have  shown  the  reduced  matrix  parser  to  be  cue  of  the  fastest 
practical  parsers  known. 
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Appendix  I 


< 

ENT  of  Test  Grammars 


Arithmetic  Expression  Grammar  “A?” 


S  :  :=  E 

E  ::=  T 

j  +  T 

I  ~  T 

J  E  +  T 

|  E  -  T 

T  :  :=  P 

]  T  *  P 

I  1  /  P 

P  : : =  id 

I  (  B  ) 


Keren  dak*  s  Grammar  "G11 


5 0  : :  =  S 1  A  S 1  E 

51  : :  =  a  c 
A  :  :=  fc 


I  c 

E  : :  -  c 

l  a 

C  :  :=  C  e 

J  e 
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Pager’s  Grammar  HGM 


s° 

«  ♦ 

•  •• 

s 

1 

S 

«»  »  — 

e 1 

X 

a 1 

J 

e2 

X 

a2 

1 

e3 

X 

a3 

1 

Ri 

a1 

b 

1 

H2 

a2 

b 

i 

If  3 

a3 

b 

H1 

m  +  — » 

<9  • 

c1 

a 1 

H2 

•  9  — ■» 

c2 

a  2 

h  3 

•m  • 

*  • 

c3 

a3 

SUSIE  Grammar 


<PBOGRAM>  ::=  <SCOPE_ELCCK>  EOF 
<SCOPE_BLOCK>  <  PECLA  PAT  ION  S>  <STATEMENTS> 

J  <  S  T  AT  E  M  E  N  T  S  > 


<DECLARATIONS>  :  : 
<  DECLAP AT 10  N_LIST> 

< DECLARATION}  ::  = 


1 


DECLARE  < D ECLA RATIO N_L IS T>  END  ; 
<DECLARATICN> 

]  < DECLARATION_LIST>  <EECLARATION> 

TYPE  <ID>  =  <TYPE>  ; 

<TYPE>  :  <IDENTIFIER>  ; 
<PROCEDORE_HEAD>  ;  <SC0PE_8L0CK>  END 


<TYPE> 


<ID> 

1  (  <IDENTIFIERS>  ) 

|  POINTER  TO  <ID> 

J  {  <CG  NSTANT>  TO  <CONSTANT>  ) 

|  ARRAY  (  <TYPE_LIST>  )  OF  <TYPE> 
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<IDENTIFIERS> 


<  ID> 


j  <IDENTIFIERS>  ,  <ID> 
<CONSTANT>  :  :  =  <ADBING_CPERATOR>  <  N  U  H  D  E  R  > 

|  <NUMEER> 


<T  YPE_LI ST>  = 

<TY  PE> 

) 

<TYPE_LIST>  ,  <TYPE> 

<PROCEDURE_HEAD> 

::=  PROCEDURE  <ID>  {  <DECLARAT IOM_  LIST  >  ) 

RETURNS  <TYPE> 

|  PROCEDURE  <ID>  RETURNS  <TYPE> 

< STATE HE NT S> 

<STATEHENT> 

|  <STATEKENTS>  <STATEMENT> 


<STATEMENT>  :  r= 

EXIT  <BLOCK_LABEL>  <CONDITION>  ; 

! 

EXIT  <BL.OCK_LAEEL>  ; 

1 

REPEAT  <BLCCK_LABEL>  <CONDITION>  ; 

1 

REPEAT  <BLOCK_LABEL>  ; 

I 

RETURN  RITE  <EXPRESSICN>  <CCNDITION>  ; 

I 

RETURN  WITH  <EXPRESSION>  ; 

J 

CASE  <TYPE>  TAG  <EXPRESSICN>  ;  <ALT  ERNATI VSS> 

END  ; 

1 

<BLOCK_LAEEL>  BEGIN  < SCOP E_E LCC K>  END 

<BLOCK_LAEEL>  ; 

I 

<EXPRESSION>  ; 

1 

• 

t 

<BLOCK_LAE  EL>  :: 

=  <  < ID>  > 

<CONDITION>  :  :  = 

WHEN  <EXPRESSION> 

1 

UNLESS  < EXPRESSIO  N> 

ALTERNATIVES >  <CAS E_LAEEL>  <STATEMENTS>  <A LTER NATI VES> 

|  ELSE  :  <STATEMENTS> 
j  <CASE_LAEEL>  <ALTERNATIVES> 
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|  <CASE_LAEEL>  <STATEMENTS> 
<CASE_LABEL>  ::=  <TD>  : 

|  <NUMBER>  : 

<  EXPRESSED  N>  : :  =  <SUM> 


|  < S U M >  <RELATIOM>  <SUM> 

I  <PRIMA  RY>  :=  <  EXPRESSIC  N> 

<SOM> 

<ADDI NG_0 PERATO R>  <TEPM> 

! 

<TERM> 

1 

<SUM>  <ADDI NG_OPERATOR>  <TERE > 

<TER!S>  ::= 

<PRI KARY> 

1 

<TERE>  <  MULT  IPX YING_OPERATOR  >  <FRIMARY> 

1 

<NUEEER> 

1 

(  <EXPRESSICN>  ) 

< PRIM ARY > 

< ID> 

1  <FRIKABY>  a) 

< PHI EASY >  {  <EXPRESSIQNS>  ) 

<  EXPR  ESSIONS>  <EXPRESSICN> 

1  <EXPRESSIGNS>  ,  <EXPRESSICN> 
<ID>  <IDENTIFIER> 

J  "  <STR I NG>  " 

<ADDING  OPERATOR>  + 


<RELATION> 

1  - 

:  :=  < 

1  ^ 

1  > 

!  -  = 

it  ii 

V  A 

<MULT. I  PLYING  OPERATOR>  : :=  * 
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|  MOD 


V 
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> 


-134- 


Append ix  II 


Parser  Table  I  implementation  Considerations 

Since  the  IjEK/360  is  oriented  toward  an  8  bit  byte  for  ease 


of  data 

access,  we  choose  to  implement  the  table  entries  as 

a  rrays 

of  bytes.  This  width  is  a  logical  choice  since  the 

maximum 

parse  state  number  "g"  is  less  than  23  for  the  test 

grammars,  as  is  n=p+g.  (If  p  +  q  had  exceeded  28,  the  array  could 
have  been  implemented  in  halfwords.)  Thus  the  following  is  stored 


in  byte 

arrays: 

1) 

The  transition  table; 

2) 

The  Indirect  address  vectors  for  the  transition  and 

error  tables; 

as  well  as  certain  auxilliary  information  needed  in  a  running 


parser: 

3) 

The  symbol  index  into  the  grammar  vocabulary  of  the 

LHS  of  each  production.  (This  is  used  to  resume  the 

parse  after  a  production  is  applied.) 

4) 

The  length  of  the  RHS  of  each  production. 

(This  is  used  in  the  stack  popping  after  each 

production  is  applied.) 

5) 

The  symbol  index  into  the  grammar  vocabulary  of 

each  non-apply  parse  state.  (This  is  used  for  error 

diagnostic  purposes.) 

The  dimension  of  the  arrays  3)  and  4)  is  the  number  of 
productions;  that  for  5)  is  the  number  of  states. 
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The  reduced  error  detection  array  E*  is  stored  as  an  array  of 

bits,  packed  by  rows  eight  bits  per  byte,  with  each  row  of  the 

< 

array  beginning  on  an  integral  byte  boundary.  Because  errors  are 
detected  in  the  input  text  stream  only  during  the  inspection  of 
terminal  symbols,  only  the  terminal  portion  of  the  array  is 
stored . 

The  read/look-ahead  flag  array  is  packed  eight  bits  per  byte, 
and  stored  by  rows  in  a  byte  array,  with  each  row  starting  on  an 
integral  byte  boundary.  This  choice  of  storage  representation 
for  the  flags  is  very  compact,  but  it  has  potential  speed 
implications  in  the  accessing  of  any  one  bit. 
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Reassigned  Reduced  Parser  Tab le  Implementation  Considerations 


Consider  the  impleme 
4.3  in  a  functioning 
considerations  discussed 
parsing  table  data  struct 


These  data  structures  are 

1) 

The  reassigned  r 

2) 

The  apply  produc 

3) 

The  reduced  erro 

4) 

The  indirect  add 

error  matrices; 

5) 

The  base  address 

transition  matri 

and 

the  a 

uxilliary  inform 

6) 

The  production  L 

apply  production 

7) 

The  production  R 

8) 

The  state  access 

The 

s  torn 

ge  space  cccupie 

deter  mine 

d  experimentally 

the 

last 

five  items  ma 

the  unreduced  unassigned 


ntation  of  the  table 
parser.  Following 
in  chapter  4  and 
ures  are  implemented 

-m 

m 

educed  transition  mat 
tion  table  data  list; 
r  attribute  matrix; 
ress  vectors  for  the 

vectors  for  the  ED  e 

x; 

ation  needed  by  a  run 
HS  vocabulary  symbol 
s ta  te ; 

HS  length; 

symbol  vocabulary  in 
d  by  the  first  three 
by  grammar  process.in 
y  be  determined  from 
SLR  ( 1)  flow  table  for 


structure  of  Figure 
the  implementation 
in  Appendix  II,  the 
in  byte  arrays. 

r  ix ; 

transition  and 

n co ding  of  the 

i 

ning  parser: 
index  for  each 

dex . 

items  above  must  be 
g;  the  dimension  of 
the  grammar  or  from 
the  grammar. 


Consider  a  straightforward 
matrix  structure  by  storing  each 
with  the  optimization  that  th 


imple mentation  of 

the 

tr  an 

si tion 

transition  entry 

in 

one 

byte. 

e  read/look-ahead 

flag 

is 

packed 
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within  the  byte,  as  is  the  flag  that  distinguishes  an  apply 
transition  from  a  non-apply  transition.  (These  optimizations 
cause  no  problems  since  there  are  fewer  that  26  transitions  in 

any  row  or  column  of  the  test  grammar  flow  tables.  There  are 

probably  few  grammars  of  realistic  size  that  have  more  than  2& 

specified  different  transitions  in  any  row  or  any  column.  If 

such  grammars  are  processed,  a  larger  data  width  for  storage  may 
be  used.)  The  data  storage  space  for  the  transition  matrix  in 
bytes  is  thus  the  number  of  matrix  entries.  The  reduced  error 
detection  matrix  is  packed,  as  previously  mentioned,  eight  bits 
per  byte.  The  apply  production  table  list  contains  one 
production  number  per  byte,  since  there  are  fewer  than  28 
productions  in  any  one  of  the  test  grammars. 

From  the  above,  the  size  of  the  reassigned  reduced  parser 
tables  in  bytes  may  be  computed: 

SIZE  =Sl+S2+S3+4*  Estates  +  2  *  ^symbols 
+  2  *  #prods  +  tterminals 

where  Si  =  fentries  In  the  assigned  reduced  matrix  PI" 

(from  table  4.13,  using  the  smallest  reduced  M” ) 

52  =  size  of  the  reduced  packed  error  attribute 

matrix  E*  (from  table  4.12) 

53  -  number  of  entries  in  the  overlapped  apply 

production  data  list 

tstates  =  the  number  of  non-apply  (read)  states 
flsyrabols  -  the  number  of  vocabulary  symbols 
#prods  =  the  number  of  grammar  productions 
tterminals  -  the  number  of  vocabulary  terminal 

symbols . 
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Appendix  IV 

Factorized  Parser  Table  Implementation  Considerations 

Consider  a  straightforward  implementation  of  the  TNT  and  HINT 
factored  machines  in  a  working  parser,  using  the  same  assumptions 
as  employed  in  the  unfactored  parser  table  implementation.  For 
the  TNT  assigned  machine  the  following  data  and  pointers  are 
required : 

1)  The  T  machine  and  the  NT  machine  reduced  matrices; 

2)  The  reduced  error  detection  matrix; 

3)  The  apply  production  table  lists  for  the  T  and  the 
NT  machines; 

4)  The  parser  matrices  indirect  address  vector  entries 
(2  *  Estates  +  #terminals  +  #non-ter m inals) ; 

5)  The  error  detection  matrix  indirect  address  vectors 
(Estates  +  ^terminals) ; 

6)  The  apply  production  table  pointer  entries 
(2  *  Estates) ; 

7)  The  base  address  vector  entries  for  the  non-apply 
transitions  {^terminals  +  fnon- terminals) . 

For  the  RLNT  assigned  machine  the  required  information  is; 

1)  The  reduced  R,  L,  and  NT  machine  matrices; 

2)  The  error  detection  reduced  matrix; 

3)  The  apply  production  table  lists  for  the  R,  L, 
and  NT  machines; 

4)  The  parser  matrices  indirect  address  vector  entries 
(2  *  Estates  +  2  *  ^terminals  +  # non- term ina Is 

+  £LA  states) ; 
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5)  The  error  detection  matrix  indirect  address  vector 
entries  (Estates  +  tterminals) ; 

< 

6)  The  apply  production  table  pointer  entries 
{2  *  Estates  +  #IA  states) ; 

7)  The  base  address  vector  entries  for  non-apply 
transitions  {2  *  #terminals  +  #non- terminals) . 

<In  .  the  RLNT  machine,  net  all  transitions  in  the  LA  machine  are 
to  apply  production  states,  therefore  a  base  address  vector  roust 
be  maintained  for  it.) 

In  addition,  both  factored  machines  must  include  the 
auxilliary  information  required  by  a  running  parser:  the 

production  LHS  vocabulary  symbol  indices,  the  production  RHS 
lengths,  and  the  state  access  symbol  vocabulary  indices.  If  the 
states  of  the  unfactored  unreduced  machine  M*  are  ordered  such 
that  all  states  containing  look-ahead  transitions  appear  before 
those  read  states  that  do  not,  the  state  access  symbols  for  the 
factored  LA  machine  states  will  be  the  same  as  those  for  the 
factored  R  machine  states.  Thus  the  total  number  of  auxilliary 
information  entries  is 

2  *  #prods  +  ^states. 
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Cost  Eenefit  of  the  Matrix  Parser  in  XCOM  at  UTCC 


< 


Consider  the  substitution  of 


reduced 

tables  fo 
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compiler 

for 

XPL. 

The 

parser 
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parser 
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t  h 

e  net 

comp il 

bytes . 

The  compiler  also  requires 
buffer  space,  and  workspace  used  by 
a  50K  area  would  probably  be 
requirements,  the  compiler  is  often 
bytes  in  order  to  avoid  unnecessary 
during  compilation. 


the 

matrix 

parser 

and  tab 

for 

the  ma 
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r  recove 
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er  size  of  app 
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enough 
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execu 
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trix  tables 
yed  for 
ry)  .  The  ta 
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roxi mate ly 

orage  space, 
system.  W 
tisfy  the  s 
region  of 
the  string  s 


Based  on  a  200K  byte  compilation  region,  the  cost  cf  u 
the  XPL  compiler  executing  on  the  IBM  37C  model  165  at 
University  of  Toronto  Computer  Center  (UTCC)  may  be  calcul 
from  the  pricing  formula  employed  by  the  installation: 
Equivalent  Run  Time  =  CPU  time  +  I/O  wait  time 
Core  Charged  =  Core  Allocated  *  (1  +  Core  Allcca ted/5  00) 
Core  Usage  -  Equivalent  Run  Time  *  Core  Charged 
CPU  Cost  =  $8.50  per  CPU  minute 


h  a  i  n 
XCOM 
is 
LALR 
bles 
as 
not 
100K 

I/O 
hi  le 
pace 
200  K 
pace 

sing 

the 

ated 
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Core  Osage  =  SI. 05  per  hundred  kilobyte-minutes 
Total  Cost  =  CPU  Cost  +  Core  Usage  Cost  +  I/O  Cost 
The  total  I/O  wait  time  and  the  I/O  cost  charges  are  not 
evaluated  here.  Thus  the  equivalent  run  time  is  composed  of  CPU 
time  for  our  purposes.  The  total  job  cost  would  thus  be  SB. 50 
CPU  plus  $2.95  core  costs,  or  $13.45  per  CPU  compilation  minute. 

Based  on  this  evaluation,  the  15%  savings  amounts  to 
approximately  S2.00  per  compilation  CPU  minute,  excluding  I/O 
charges,  when  the  matrix  parser  is  employed.  Even  replacing  a 
LALR  parser  in  XCCM  would  save  7%  of  the  compilation  time,  or 
approximately  $0.95  per  compilation  CPU  minute.  It  is  clear  that 
for  the  duration  of  a  one  term  or  a  two  terra  course  in  compiler 
writing  technology  that  the  accumulated  savings  could  be 
substantial. 
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XPL  Scanner  Grammar  ENF 


<TOKEN 

SET> 

•  •  — 

•  » 

<SL 1 >  EOF 

<SL  1  > 

♦  *  zz. 

♦  » 

<SFPARATOR_ELANK> 

J 

<SL  1  > 

<SEPA  RATOR_ELA  N 

K> 

I 

<SL2> 

<SEPARATOR_BLAN 

K> 

! 

<  S  L  3  > 

<SE  PA  RATOR_  ELAN 

K> 

1 

<SL4> 

<SE PA RATO R_ ELAN 

K> 

<SL2> 

m  •  — 

«•  * 

<1 DENT IFIE  R> 

! 

<SL  1  > 

CIDENTI FI E  R  > 

1 

<SL3> 

<IDENTIFIER> 

1 

<SL4> 

<IDENTI FIFR> 

<SL3> 

•  *  — — 

>•  » 

<N!JMBER> 

1 

<SL  1  > 

<NUHEER> 

1 

<SL4> 

<NUHBES> 

<SL4> 

•  •  — *- 

•  • 

<STRIKG> 

1 

<SL  1  > 

<STRING> 

) 

<  S  L  2  > 

<ST  RING> 

1 

<SL3> 

<ST  RING> 

<NUMBER>  :: 

=  <INTEGER> 

|  <BIT  Q OOTE>  <P IT  LIST>  ” 
<INTEGER>  < EECI J1AL  DIGIT> 

|  <INTEGEB>  <EECIMAL  DIGIT> 
<SEPARATOR_BLANK>  <  SEPA  RATO  R> 

|  <ELANK> 

|  /*  <CHARACTERS>  */ 

<IDENTIFIER>  <LETTER> 
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|  <IDENTIFIER>  < LETTER > 

\  <IDENTIFIER>  < DECIMAL. DIGIT> 
<LETTER>  =  <LETTE  R_DIGIT> 

|  <0T HEP.  LETTER> 

<  DECIMAL  DIGIT>  <OCTAL  DIGIT> 

j  <  DI GIT  S  8 ,  9> 

<STRI NG>  <Q  UCTE>  <CHAPACTERS>  » 

!  1  1 

<CHAPACTERS>  : <CHARACTFR> 

]  <CHAPACTERS>  <CHAHACTER> 


<QUOTE>  » 

<EIT  QUOTE>  :  :  = 

ti 

<EIT  LIST>  :  :  = 

<  REX 

INTEGEP> 

1 

<EIT 

GRO  DP> 

1 

<BIT 

LIST>  <EIT  GPOU  P> 

<EIT  GROUP> 

(  1 

)  < E I N A R Y  I NTEGER> 

1 

(  2 

)  <  QUARTAL  INTEGER> 

1 

(  3 

)  <OCTAL  INTEGER> 

1 

(  4 

)  <  REX  INTEGER> 

CEINARY  INTEGEH> 

♦  • 

* 

<BINARY  DI GIT> 

1 

<E INARY  INTEGER>  <EINAP Y 

BINARY  DI GIT>  :: 

=  0 

]  1 

<QUARTAL  I NTEG ER> 

•  »  ~ 

<QUAPTAL  DI GIT> 

J  <QUARTAL  INTEGER>  <QUA  R TAL  DIGIT> 
<QDARTAL  DIGIT>  <  B I N  A  R  Y  DIGIT> 

I  2 

1  3 

<OCTAL  INTEGEE>  ::=  <OCTAL  DIGIT> 
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< OCTAL  DIG IT> 


<  HEX  INTEG  ER> 


<HEX  DIGIT> 


J  <OCTAL  IHTEGER>  <OCTAL  DIGIT> 
:=  < QUART AL  DIGIT> 

1  ^ 

|  <  D I G I T  S  5,6,7> 

:=  <  H  E  X  C I G I T  > 

j  < HEX  INTEGERS  <HEX  DIGIT> 
<BECIMAL  DIGIT> 

<LETTER  DIGIT> 
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Appendix  VII 


Example  of  Parser  Matrix  State  Reduction  by_Alqori thm  $3 


Korenjakts  grammar  KG,  given  in  Appendix  I,  is  processed  by 
DeRemer’s  SLR(1)  analysis  algorithm,  yielding  the  following 
hybrid  LR(0)-LR{1)  table: 


Symbols 


State 

1 

a 

b  c  d  e 

1  S°  S1  A  13  C 

1 

9 

1 

15,  R 

10 

1 

2,  L  2,  L  2  ,  L  7  ,  R 

1 1 

1 

9,R 

12 

t 

3  ,  R  4 ,  R 

1  4,  R 

13 

1 

8  ,  R 

10,  R 

14 

1 

1  3,R 

16, R 

+  15 

1 

13, R 

1 1, R  12, R 

16 

1 

5 ,  R  6  ,  S 

1  f  R 

sLgOl 

Parse  Table  K*  for 

Korenia^s  Grammar  ” KG M 

+  start  state 


Notes : 

a)  The  table  description  fellows  that  outlined  on  page  47, 

b)  Only  parse  (non-LR(O)  reduce)  states  are  represented  in  the 
table:  ie-  only  those  whose  state  number  is  greater  than  the 
number  of  grammar  productions  (eight) . 

c)  The  output  symbol  set  is  (Read,  Look-ahead}. 
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States  9  and  11  are  inserted  by  the  EeRemer  SLR(1)  analyzer, 
which  adds  a  zeroeth  production  S  ->  J_  s°  j_  to  the  grammar. 
These  states  are  not  required,  nor  is  the  column  headed  by  S° 
since  their  entries  are  never  accessed.  Thus  the  parser  matrix 
contains  14  states  excluding  states  9  and  11  (eight  apply  states 
and  six  parse  states).  However,  states  9  and  11,  and  the  column 
S 0  will  be  left  in  the  parser  table  for  the  purpose  of 
illustrating  the  state  reduction  by  Algorithm  #3  using  disjoint 
compa  tibility . 

Form  the  set  of  pairwise  disjoint  incompatible  states  c  (s) 
for  each  state  9<s<16  ; 

D(s)  -  {  (11),  (12,13,16),  (9),  (10,16),  (10),  (15), 

(14)  ,  (10,12)  }  , 

where  D  (9)  =  (11),  D  ( 1 0)  =  (12,13,16),  ... 

Form  the  state  decomposition  sequence  SE.C  fcy  Roy  and  Sheng*s 
method.  The  sequence  is  SEQ  =  (10,  9,  12,  14},  therefore 

#steps= ] SEQ ] =4 . 

To  produce  the  first  cover  block  B{1),  choose  the  states 
pairwise  DC  with  state  10  as  the  initial  set  R(1)  of  potentially 
maximal  states,  p  ( 1  0)  .  Thus  R  { 1 )  -  ( 9 ,  1 0 ,  1 1 ,  1 4 ,  1 5)  .  Decompose 
R(1)  by  the  remaining  states  in  the  deco mpcsi t ion  list  SEQ  (i)  , 
for  1<i<4,  according  to  Step  7  of  Algorithm  #3: 

a)  (9,10,11,14,15)  -  {9}  =  (10,11,14,15) 

b)  (10,11,14,15)  -  {14}  -  (10,11,15). 
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' 


' 


The  decomposition  by  state  12  after  step  a)  is  not  performed 

since  state  12  is  not  contained  in  R{1) .  Therefore 
B  { 1)  =  { 10,  1 1 ,  1 5)  is  the  first  DC  cover  block. 

To  produce  the  second  cover  block  E  (2)  ,  choose  for  R{1)  the 
states  pairwise  DC  with  state  9  not  covered  by  the  states  in 

B(1)  : 

*(1)  =  H{9)  “  E(1)  -  (9#  12,  13,  14,16)  . 

Decompose  R  ( 1 )  by  SEQ  (i)  for  2<i<4: 

a)  (9,  12,  13,  14,16)  -  {12}  -  (9,13,14,16). 

The  decomposition  by  state  14  is  net  performed  since  R(1)- 

D{14)=0r.  Therefore  B  (2)  -  (9,13,14,16)  is  the  second  DC  cover 

block . 

To  produce  the  third  cover  block  E  (3)  ,  choose  for  R  ( 1 )  the 
states  pairwise  DC  with  state  12  not  covered  by  the  states  in 

B  ( 1)  or  in  B  { 2)  : 

R  ( 1 )  =  D  ( 1 2)  -  B  { 1)  -  B  { 2)  =  (12). 

The  decomposition  by  state  14  is  net  performed  since  state  14  is 
not  contained  in  R(1)  .  Therefore  E  (3)  =  (12)  is  the  third  DC 
cover  block. 

Since  all  parse  states  are  now  covered  by  the  three  cover 
blocks  in  E,  the  algorithm  halts.  The  reduced  solution  DC  cover 
H"  of  the  parser  matrix  is  thus: 

B=  {  (10,11,15),  (9,13,14,16),  (12)  }. 
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The  solution  produced  above  is  theoretically  minimal  since 

the  DC  lower  bound  for  the  unreduced  machine  K*  is  three  blocks: 

' 


DCLB (H») =3, 


' 


' v  V  . 


> 


- 
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Appendix  VIII 

Roy  and  Sheng*s  Decomposition  Technique 

The  technique  of  Roy  and  Sheng  [1972]  is  a  direct  method  of 
determining  the  maximal  compatibility  classes  of  the  state  set  S 
of  an  incompletely  specified  FSM  M 1 .  Subsets  of  pairwise 
incompatibles  are  used  to  decompose  S  in  a  step-by-step  process 
into  the  maximal  compatibles.  Pauli  and  Unger  [1959]  suggested  a 
process  of  decomposing  the  state  set  S  successively  using  each  of 
the  state  pairwise  incompatibility  relations.  Their  process  may 
be  shown  to  require  "k"  steps,  where  0<k<nC2  (nC2  is  the  binomial 
coefficient).  Thus  "k"  may  greatly  exceed  the  number  of  states 
in  M*.  Roy  and  Sheng1s  method  decomposes  the  set  of  states  S  of 
H*  not  by  individual  incompatible  state  pairs,  but  by  groups  of 
them.  The  number  of  steps  in  their  process  is  always  less  than 
the  number  of  states  in  H*. 

If  state  si  €  S  is  pairwise  incompatible  with  each  of  the 
states  in  the  set  D  (si)  (where  D  (si)  =  {si1,si2, . . . ,  sim] , 
m=JD(si)  |,  and  m  <  |  H 1  | )  ,  then  the  incompatibility  relationship 
decomposes  a  set  of  states  R  by  the  following  two  rules: 

1)  R  ->  R  if  state  si  does  not  belong  to  R  or  D  (si)  is  not 
contained  in  R. 

2)  R  ->  R 1  ,  R2  if  si  belongs  to  R  and  P(s.i)  is  contained  in 
R,  where  Ri=R-(si},  and  R2=R-D  (si)  by  the  set  difference 
operation.  R1  and  R2  are  called  "residue  sets". 
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The  above  rules  are  applied  iteratively  to  all  generated 
residue  sets  using  the  set  of  all  states  S  as  the  initial  state 
set  for  B.  If  we  choose  a  suitable  sequence  of  states  (the  si 
above),  a  binary  decomposition  tree  is  developed.  At  the  end  of 
11  k”  decomposition  steps  (with  nk"  different  values  of  si)  the 
process  halts,  leaving  the  set  of  maximal  compatible  blocks  C. 
(Some  of  these  blocks  may  be  duplicated:  ie-  a  Cj  and  a  Ck  €  C 
may  exist  at  any  level  of  the  decomposition  such  that  Cj  c  Ck. 
In  this  case,  Cj  is  deleted  and  Ck  is  retained  in  the  set  C.) 

The  selection  of  a  suitable  state  decomposition  seguence  (ie- 
the  sequence  of  states  si  above)  is  performed  using  the 
distribution  table  technique  of  Boy  and  Sheng.  The  following  is 
an  algorithmic  representation  of  this  technique.  The  state 
decomposition  seguence  is  contained  in  the  array  "SEQ"  when  the 
algorithm  halts. 


Step  1)  Initialize  the  distribution  table  values: 

Distrib  (s)  =  |D(s)  ]  for  all  s  in  S. 

Step  2)  Clear  the  count  of  the  number  of  elements  in  the 
decomposi tio n  sequence  array:  J  =  0. 

Step  3)  Find  the  state  s*  of  S  having  the  largest  value  of 
Distrib(st).  If  there  is  more  than  one  such  state  s’, 
choose  the  first  one  found. 


Step  4)  If  Distrib(s’)  =  0,  then  halt. 


. 

Step  5) 


Step  5) 

Add  state  s*  to  the  state  decomposition  sequence  array: 

J  =  J  +  1  ;  SEQ  (J)  =  s'  . 

< 

Step  6) 

Clear  the  distribution  table  value  for  state  s’: 

Dist r ib  { s  1 )  =  0 . 

Step  7) 

Decrement  the  distribution  table  value  for  each  state  s 

that  is  pairwise  incompatible  with  state  s’ : 

Distrib(s)  =  MAX  (Dist  rib  (s)  -  1 , 0)  for  all  s  in  D{s1)  . 

Step  B) 

Go  to  step  3)  . 

Applying  -the  above  algorithm  to  the  pairwise  disjoint 
incompatible  states  D (s)  of  Keren jak’s  grammar  KG  in  Appendix  VII 
yields  the  staled  four  element  decomposition  sequence  in  the 
array  SEQ. 
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